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Abstract 

“Performance-based  acquisition  (PBA)”  means  an  acquisition  structured  around 
the  results  to  be  achieved  as  opposed  to  the  manner  by  which  the  work  is  to  be 
performed.  Performance-based  acquisition  is  a  results-oriented  acquisition  strategy  used 
to  achieve  innovative  solutions  in  agency  programs.  Most  military  systems  have  been 
used  for  decades,  so  future  planning  of  defense  procurement  is  critically  important. 
Performance-based  acquisition  is  a  strategic  method  to  manage  business  by  promoting 
flexibility  and  innovation  and  creating  win/win  solutions  through  effective 
communication,  organizational  goal  alignment,  and  clear  accountability  among  the  buyer, 
seller(s),  and  subcontractors. 

In  this  century,  many  governments  are  transitioning  their  acquisition  strategy 
from  traditional  methods  to  perfonnance-based  methods.  In  2000,  the  U.S.  Department 
of  Defense  set  a  goal  that  a  minimum  of  50%  of  the  service  acquisitions  would  use 
performance-based  acquisition  methods  by  2005.  Do  cultural  differences  have 
significant  importance  to  the  success  of  this  new  acquisition  method?  Applying  PBA 
methods  in  those  Western  countries  have  shown  successful  results.  What  results  can  be 
obtained  from  applying  this  new  strategy  in  non-Westem  countries?  This  study  focuses 
on  the  success  of  performance-based  acquisitions  in  non-Western  countries. 
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IMPLEMENTATION  OF  PERFORMANCE-BASED  ACQUISITION 
IN  NON- WESTERN  COUNTRIES 


I.  Introduction 


Background 


“It  is  the  policy  of  the  Department  of  Defense  that,  in  order  to 
maximize  performance,  innovation  and  competition,  often  at  a  savings, 
performance  based  strategies  for  the  acquisition  of  services  is  to  be  used 
wherever  possible.  While  not  all  acquisitions  for  services  can  be 
conducted  in  a  performance-based  manner,  the  vast  majority  can.  Those 
cases  in  which  performance-based  strategies  are  not  employed  should 
become  the  exception.  In  order  to  ensure  that  the  Department  continually 
realizes  these  savings  and  performance  gains,  the  DoD  establishes,  at  a 
minimum,  that  50  percent  of  service  acquisitions,  measured  in  both  dollars 
and  actions,  are  to  be  performance-based  by  year  2005.  ’’ 

-  Under  Secretary  of  Defense,  Acquisition,  Technology  & 
Logistics  (USD  (AT&L),  April  5,  2000). 

The  trend  toward  the  use  of  perfonnance-based  acquisitions  (PBA)  for  U.S. 
government  programs  began  in  the  late  1970’s  and  early  1980’s  and  has  accelerated  ever 
since.  In  September  1979,  the  U.S.  Air  Force  adopted  a  comprehensive  perfonnance- 
based  approach  to  contracting  for  base  support  services.  Soon  after,  the  Office  of  Federal 
Procurement  Policy  (OFPP)  adopted  the  performance-based  approach  for  government¬ 
wide  use.  In  1991,  the  OFPP  issued  a  policy  letter  in  response  to  growing  concerns  about 
the  amount  of  money  that  agencies  were  spending  to  buy  services  and  the  quality  of  the 
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services  they  were  receiving.  In  1997,  the  Federal  Acquisition  Regulation  (FAR)  was 
amended  to  include  formalized  rules  for  PBA.  Finally,  in  2001,  the  Office  of 
Management  and  Budget  (OMB)  issued  a  mandate  that  directed  government  agencies  to 
increase  their  use  of  performance-based  techniques  on  an  aggressive  schedule  (Table  1). 


Table  1.  Office  of  Management  and  Budget  Goals 


Fiscal  Year  2002 

20% 

Fiscal  Year  2003 

30% 

Fiscal  Year  2004 

40% 

Fiscal  Year  2005 

40%  (Changed  from  50%  to  40%  by  OFPP) 

Fiscal  Year  2006 

40% 

Source:  OMB  Memorandum  M-0 1-1 1  (2/14/01  memo)  and  M-01-15  (3/9/01  memo) 


Since  the  publication  of  the  Office  of  Federal  Procurement  Policy’s  (OFPP) 
policy  letter  91-2,  Service  Contracting,  on  April  15,  1991,  Performance-based  services 
acquisition  (PBSA)  has  been  the  U.S.  government’s  preferred  approach  to  service 
contracting. 


Problem  Statement 

In  this  century,  shrinking  government  budgets  encourage  agencies  to  find  more 
cost-effective  ways  to  obtain  services.  One  way  to  do  this  is  to  focus  on  the  results  to  be 
achieved  (e.g.  “statements  of  objectives”)  as  opposed  to  the  manner  by  which  the  work  is 
to  be  perfonned  (e.g.  “statements  of  work”).  Performance-based  acquisition  (PBA)  is  a 
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relatively  new  contracting  method  in  the  acquisition  area  designed  to  achieve  this  focus 
shift.  Theoretically,  programs  using  PBA  methods  ought  to  achieve  greater  perfonnance 
at  a  lower  cost  than  they  would  have  otherwise.  For  instance,  current  expectations  are 
that  the  F-35’s  life-cycle  cost  will  be  about  20%  lower  when  compared  to  that  of  legacy 
systems  such  as  F/A-16  and  F/A-18  (Sols,  Nowick,  Verma,  2007).  Additionally,  the 
United  Kingdom’s  Royal  Air  Force  (RAF)  is  expecting  to  reduce  the  maintenance  costs 
of  E-3D  Sentry  Airborne  Warning  and  Control  System  (AWACS)  aircraft  by  12  percent 
over  previous  similar  systems.  Finally,  other  countries  such  as  Spain,  Argentina,  and 
Chile  have  been  using  PBA  with  the  expectation  of  cost  savings  and  perfonnance 
improvements. 

A  key  question,  however,  is  whether  all  countries  may  apply  PBA  methods 
equally  well,  or  in  the  same  manner.  Given  cultural  and  socioeconomic  differences 
between  countries,  is  it  even  appropriate  to  use  PBA  in  non-Western  countries?  Perhaps 
applying  PBA  methods  in  different  countries  will  result  in  different  outcomes.  Before 
non-Westem  countries  (such  as  Turkey)  attempt  to  adopt  PBA  methods,  acquisition 
professionals  ought  to  analyze  whether  cultural  and  socioeconomic  differences  could 
hinder  successful  implementation. 

The  universality  of  business  practices  and  employee  attitudes  has  been  the  focus 
of  most  cross-national  studies  in  recent  decades.  Since  perfonnance-based  acquisition  is 
relational  and  interactive  compared  to  traditional  acquisition  methods,  understanding 
cultural  differences  may  give  acquisition  professionals  insight  for  developing  and 
managing  the  PBA  environment  in  different  countries.  A  better  understanding  of  the 
effectiveness  of  managerial  practices  and  the  differences  in  employee  attitudes  could 
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make  management  of  international  organizations  more  effective  (Ronen  &  Kraut,  1977). 
Countries  or  nations  can  be  clustered  by  comparing  on  general  attitudes  towards  work,  or 
by  using  variables  such  as  language,  religion,  or  level  of  industrialization  (Ronen  & 
Shenkar,  1985).  Clusters  can  be  generalized  to  other  nations  or  countries.  Clusters  can 
also  be  used  to  facilitate  the  success  of  PBA  environment  by  explaining  the  variance  in 
work  goals  and  managerial  attitudes.  After  a  thorough  literature  review,  Ronen  and 
Kraut  (1977)  found  eight  cultural  clusters  based  on  variables  such  as  work  goals,  need 
deficiency,  fulfillment,  and  job  satisfaction.  Work  goals  are  less  constrained  and  they 
best  represent  the  cultural  environment  of  individuals,  while  work  values  are  linked  to 
cultural  milieu  and  job  individual’s  job  behavior  (Ronen  &  Shenkar,  1985).  Additionally, 
each  country  has  different  regulations,  different  steps  to  execute  the  contracts,  and 
different  thresholds  according  to  time,  length,  and  urgency  of  the  contract.  To  add  to  the 
complexity  of  this  issue,  many  countries  from  different  cultural  clusters  are  acquiring  the 
same  weapon  systems  through  joint  venture  acquisitions  such  as  F-35  Joint  Strike  Fighter 
(JSF)  and  A-400M  military  transport  aircraft.  How  will  this  affect  the  success  or  failure 
of  the  acquisitions? 

Research  Objectives/Research  Questions  &  Hypotheses 

The  objective  of  this  research  is  to  examine  the  results  of  the  perfonnance-based 
acquisition  (PBA)  contracts  in  various  countries  and  assess  whether  it  would  be 
beneficial  to  transition  to  PBA  in  countries  that  are  currently  using  traditional  (non-PB A) 
acquisition  methods.  Specifically,  this  research  will  attempt  to  identify:  1)  key  factors  for 
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successful  PBA  implementation,  and  2)  problems  and  challenges  to  successful  PBA 
implementation.  The  study  is  designed  to  help  predict  the  success  or  failure  of  PBA 
methods  in  countries  that  are  still  using  traditional  (non-PBA)  acquisition  methods.  To 
meet  this  research  objective  this  study  will  focus  on  the  following  question: 

'r  Can  performance-based  acquisition  methods  be  successfully  used  in  non- 
western  countries  that  are  currently  using  traditional  acquisition  methods? 

To  help  to  develop  the  main  research  question,  this  study  will  address  the 
following  investigative  questions: 

'r  Ql:  Which  variables  should  be  considered  to  monitor  whether  a  contract 
is  successful  or  not? 

'r  Q2:  Do  cultural  differences  affect  the  success  of  PBA  implementation? 

Research  Focus 

The  focus  of  this  research  is  to  determine  the  key  metrics  for  perfonnance-based 
acquisition  contracts  and  examine  the  applicability  of  the  PBA  contracts  in  different 
country’s  acquisition  systems’  (especially  Turkey).  In  order  to  discover  actual 
consequences  of  the  PBA  contracts  this  study  will  look  into  the  completed  contracts’ 
results  from  different  cultural  clusters.  What  variables  affect  the  PBA  process  and  how 
those  variables  are  implemented  in  different  countries  will  be  the  focus  of  this  study. 
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Methodology 


This  study  will  use  qualitative  methods,  such  as  the  grounded  theory  approach  to 
theory  development  (Glaser  &  Straus,  1967),  and  the  case  study  method  of  research  (Yin, 
2003).  Using  guidance  from  qualitative  research  literature,  a  semi-structured 
questionnaire  was  used  as  the  data  collection  tool  for  the  target  question  of  the  study. 

This  questionnaire  includes  seven  open-ended  exploratory  discussion  questions.  These 
questions  were  asked  to  program  managers,  contracting  officers,  project  managers,  and 
liaison  officers.  Data  gathered  from  the  interviews  were  transcribed  into  a  Microsoft 
Excel  spreadsheet  and  each  sentence  given  a  code  in  order  to  enable  prediction  and 
explanation  of  the  answers.  The  results  which  will  emerge  from  the  categorized 
sentences  will  be  analyzed  using  open  coding. 

Assumptions/Limitations 

Although  this  study  attempts  to  research  whether  perfonnance-based  contracts 
can  be  used  or  not  in  different  countries,  there  are  some  gaps  and  limitations  in  the  PBA 
field.  The  major  limitations  on  this  study  are  the  difficulty  of  determining  the  success  of 
the  contract  and  collection  of  data  on  the  PBA  field.  In  order  to  set  the  success  points  of 
a  contract  there  should  be  reference  points  or  cost  estimates  about  the  contract.  No  one 
can  say  that  a  contract  is  successful  or  not  by  just  looking  at  the  cost  of  the  contract.  The 
second  limitation  is  the  difficulty  of  collecting  data  about  the  contracts.  As  the  PBA 
contracts  are  “structured  around  the  results  to  be  achieved  as  opposed  to  the  manner  by 
which  the  work  is  to  be  performed,”  those  contracts  are  mostly  huge  contracts  consisting 
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of  hundreds,  and  sometimes  thousands  of  pages  (FAR  Subpart  2.101).  Performance- 
based  acquisition  is  a  new  strategy  in  the  acquisition  field  so  implementing  the 
perfonnance  is  another  difficulty  for  this  study. 

Implications  and  Summary 

This  study  will  provide  insight  to  different  countries’  acquisition  officers  about  the 
applicability  of  the  PBA  in  their  acquisitions.  Because  of  the  diminishing  budgets,  increasing 
defense  expenditures,  and  goal  oriented  acquisitions  focused  on  the  results,  agencies  are 
transitioning  their  acquisition  strategy  into  PBA.  Detennining  the  factors  that  are  leading  to 
success  in  acquisition  enviromnent  will  create  rigorous  enviromnent  for  PBA.  Detennining 
the  factors  of  the  applicability  of  this  new  strategy  will  also  make  the  acquisition  officers 
decide  whether  to  continue  with  the  PBA  or  not. 

This  chapter  provided  a  brief  introduction  to  perfonnance-based  acquisition  and 
articulated  the  research  problem  addressed  by  the  study.  A  limited  background 
concerning  the  study  was  presented  and  the  basic  methodology  applied  to  the  research 
was  discussed.  Chapter  2  provides  a  detailed  literature  review  of  applications  of  PBA, 
the  importance  of  implementing  PBA,  and  cultural  issues  related  to  PBA.  Chapter  3 
presents  a  full  description  of  the  methodology  used  for  data  collection  and  data  analysis. 
Chapter  4  illustrates  the  results  of  this  study  and  presents  the  analysis  of  data  and 
findings.  Finally,  Chapter  5  provides  the  resulting  conclusions  of  the  study  and  outlines 
the  limitations  and  recommendations  for  future  research. 
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II.  Literature  Review 


Introduction 

This  chapter  reviews  the  literature  relevant  to  the  creation,  implementation,  and 
evaluation  of  performance-based  acquisition  (PBA).  Because  the  U.S.  Federal 
Government  has  mandated  that  agencies  and  departments  comply  with  performance- 
based  guidelines  and  requirements,  particularly  those  outlined  in  the  Federal  Acquisition 
Regulation  (FAR)  and  agency  supplements,  understanding  and  using  performance-based 
acquisition  methods  is  now  required  of  acquisition  professionals.  The  chapter  begins 
with  a  definition  of  performance-based  acquisition  and  then  provides  an  overview  of 
Federal  acquisition  regulations,  laws,  and  policies.  Next,  the  chapter  explains  the 
essential  elements  of  performance-based  acquisition.  After  discussing  how  cultural 
differences  may  affect  the  implementation  of  performance-based  acquisitions  outside  of 
the  United  States,  the  chapter  ends  with  a  discussion  of  the  benefits  of  PBA. 

Definitions,  Federal  Regulations,  Law,  and  Policies 

According  to  the  Federal  Acquisition  Regulation  (FAR)  Subpart  2.101, 
“Performance-based  acquisition  (PBA)”  means  an  acquisition  structured  around  the 
results  to  be  achieved  as  opposed  to  the  manner  by  which  the  work  is  to  be  performed. 
“Performance  Work  Statement  (PWS)”  means  a  statement  of  work  for  performance- 
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based  acquisitions  that  describes  the  required  results  in  clear,  specific,  and  objective 
tenns  with  measurable  outcomes. 

The  U.S.  Department  of  Defense  (DoD)  spends  billions  of  dollars  every  year  for 
acquisition  of  new  major  systems,  and  for  the  continuing  maintenance,  supply  and 
support  materials,  transportation,  construction,  and  employment  of  existing  systems. 
According  to  the  “National  Defense  Budget  Estimates  for  FY  2009”  book,  DoD  has 
planned  to  spend  $104.22  Billion  for  “Procurement”  and  $180.42  Billion  for  “Operation 
&  Maintenance”  (NDBE  FY2009,  2008).  Because  there  is  a  great  deal  of  political 
oversight  of  their  spending  from  Congress,  government  agencies  have  incentive  to  try  to 
control  their  costs  and  improve  their  acquisition  outcomes.  During  the  last  couple  of 
decades,  there  have  been  many  changes  in  the  U.S.  government  acquisition  system 
designed  to  increase  its  efficiency.  One  such  change  is  the  shift  toward  performance- 
based  acquisitions  for  services. 

Since  the  publication  of  the  Office  of  Federal  Procurement  Policy’s  (OFPP) 
policy  letter  91-2,  Sendee  Contracting,  on  April  15,  1991,  Performance-based  services 
acquisition  (PBSA)  has  been  the  government’s  preferred  approach  to  service  contracting. 
It  requires  specification  of  the  results  that  contractors  must  produce  instead  of  the 
processes  that  they  must  use.  Acquisition  managers,  working-level  agency  acquisition 
personnel  and  contractors  have  spent  a  lot  of  energy  on  PBSA  since  1991.  The  Federal 
Acquisition  Regulation  provides  that  when  using  PBSA,  agencies  must  specify  the 
service  results  (outputs,  outcomes)  they  want  in  “clear,  specific,  and  objective  terms  with 
measurable  outcomes.”  They  must  prepare  performance  work  statements  and  quality 
assurance  surveillance  plans,  use  perfonnance  incentives  when  appropriate,  and  inspect 
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and  compensate  contractors  based  on  their  work  products  rather  than  their  work  processes 
(Edwards  et.al,  2007). 

The  1991  OFPP  policy  letter  was  a  response  to  growing  concerns  about  the  huge 
amounts  of  money  that  agencies  were  spending  to  buy  services  and  the  perceived  low 
quality  of  the  services  they  were  receiving.  Agencies  were  slow  to  respond  to  the  policy 
letter.  Although  the  letter  called  for  FAR  implementation  before  the  end  of  1991,  it  was 
not  until  1997  that  the  FAR  was  amended  to  include  rules  for  PBS  A.  Since  then, 
agencies  have  tried  to  use  the  technique,  but  the  results  have  been  disappointing. 
Government  acquisition  officials  and  industry  representatives  have  expressed  doubts 
about  the  success  of  PBSA.  In  July  2003,  an  interagency  team  assembled  by  OFPP 
recommended  additional  minor  changes  to  the  FAR,  which  were  published  in  December 
(Edwards  et  al.,  2007). 

Federal  Regulations 

Federal  Acquisition  Regulation  (FAR)  Part  37  provides  the  policies  and 
procedures  for  service  contracting  and  performance-based  acquisition  (PBA).  FAR 
Subpart  37.101  defines  a  service  contract  as  “a  contract  that  directly  engages  the  time  and 
effort  of  a  contractor  whose  primary  purpose  is  to  perform  an  identifiable  task  rather  than 
to  furnish  an  end  item  of  supply.”  This  part  also  notes  that  service  contracts  may  be 
either  a  nonpersonal  or  personal  contract,  and  can  also  cover  services  perfonned  by  either 
professional  or  nonprofessional  personnel  whether  on  an  individual  or  organizational 
basis. 
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Importantly,  the  FAR  indicates  that  perfonnance-based  acquisition  is  the 
preferred  method  for  acquiring  services  according  to  “Public  Law  106-398,  section  821.” 
FAR  37.102  obligate  agencies  to  use  performance-based  acquisition  methods  to  the 
maximum  extent  practicable  when  acquiring  services,  excluding: 

(1)  Architect-engineer  services  acquired  in  accordance  with  40  U.S.C.  1101, 

(2)  Construction, 

(3)  Utility  services, 

(4)  Services  that  are  incidental  to  supply  purchases. 

In  addition,  FAR  Subpart  37.102  establishes  that  the  agencies  must  use  the 
following  order  of  precedence  when  acquiring  services: 

(1)  A  firm-fixed  price  performance-based  contract  or  task  order. 

(2)  A  performance-based  contract  or  task  order  that  is  not  firm-fixed  price. 

(3)  A  contract  or  task  order  that  is  not  perfonnance-based. 

This  order  of  precedence  shows  that  DoD  gives  top  priority  to  PBA  contracts  and 
preferably  using  finn-fixed  price.  The  last  resort  for  awarding  contracts  or  task  orders  is 
non-performance  based  acquisition. 

Finally,  FAR  Subpart  37. 106  further  prescribes  the  policies  and  procedures  for 
acquiring  services  using  PBA  methods.  This  part  indicates  that  performance-based 
contracts  for  services  shall  include  the  following: 

(1)  A  performance  work  statement  (PWS); 

(2)  Measurable  performance  standards  (i.e.,  in  terms  of  quality,  timeliness, 
quantity,  etc.)  and  the  method  of  assessing  contractor  perfonnance  against  performance 
standards;  and 
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(3)  Performance  incentives  where  appropriate.  When  used,  the  performance 
incentives  shall  correspond  to  the  perfonnance  standards  set  forth  in  the  contract. 

DoD  Guidance  -  Seven  Steps  to  Performance-Based  Services  Acquisition 

Over  the  last  decade  and  a  half,  innovators  in  Congress  and  the  executive  branch 
have  refonned  the  laws  and  policies  that  govern  Federal  acquisition.  Among  the  most 
important  of  these  reforms  are  the  Government  Performance  and  Results  Act  of  1993,  the 
Federal  Acquisition  Streamlining  Act  of  1994  (FAS A),  and  the  Clinger-Cohen  Act  of 
1996.  All  of  these  laws  send  an  important  message  about  perfonnance  in  federal 
programs  and  acquisitions.  The  aforementioned  OFPP  Policy  Letter  91-2,  Service 
Contracting,”  established  that: 

It  is  the  policy  of  the  Federal  Government  that  (1)  agencies  use  performance- 
based  contracting  methods  to  the  maximum  extent  practicable  when  acquiring 
services,  and  (2)  agencies  carefully  select  acquisition  and  contract  administration 
strategies,  methods,  and  techniques  that  best  accommodate  the  requirements. 

The  intent  is  for  agencies  to  describe  their  needs  in  terms  of  what  is  to  be 
achieved,  not  how  it  is  to  be  done.  These  policies  have  also  been  incorporated  in  the 
Federal  Acquisition  Regulation  Subpart  37.6  (Perfonnance-Based  Contracting). 

Performance-based  service  acquisition  has  many  benefits.  These  benefits  are 
listed  in  Table  2: 
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Table  2.  Benefits  of  Performance-Based  Acquisition 


1 

Increased  likelihood  of  meeting  mission  needs 

2 

Focus  on  intended  results,  not  process 

3 

Better  value  and  enhanced  performance 

4 

Less  perfonnance  risk 

5 

No  detailed  specification  or  process  description  needed 

6 

Contractor  flexibility  in  proposing  solution 

7 

Better  competition:  not  just  contractors,  but  solutions 

8 

Contractor  buy-in  and  shared  interests 

9 

Shared  incentives  pennit  innovation  and  cost  effectiveness 

10 

Less  likelihood  of  a  successful  protest 

11 

Surveillance:  less  frequent,  more  meaningful 

12 

Results  documented  for  Government  Perfonnance  and  Results  Act  reporting, 
as  by-product  of  acquisition 

13 

Variety  of  solutions  from  which  to  choose 

(Source:  www.acqnet.gov/comp/seven_steps) 


The  federal  acquisition  workforce  has  not  fully  embraced  performance-based 
acquisition  (www.acqnet.gov).  The  main  reason  is  that  traditional  “acquisition  think”  is 
entrenched  in  the  workforce  of  the  acquisition  staff.  Performance-based  acquisition  is  a 
collective  responsibility  that  involves  representatives  from  budget,  technical,  contracting, 
logistics,  legal,  and  program  offices.  It  is  difficult  for  an  agency  to  assemble  a  team  of 
people  who  together  have  the  knowledge  to  drive  such  an  acquisition  through  to 
successful  contract  perfonnance.  Performance-based  service  acquisition  can  be  daunting, 
with  its  discussion  of  work  breakdown  structures,  quality  assurance  plans,  and  contractor 
surveillance. 

The  foundation  for  a  successful  acquisition  involves  a  clear  answer  to  three 
questions:  what  do  I  need,  when  do  I  need  it,  and  how  do  I  know  it  is  good  when  I  get  it? 


13 


DoD’s  virtual  guide  (www.acqnet.gov)  breaks  down  perfonnance-based  service 
acquisition  into  seven  simple  steps,  complete  with  “stories”  (case  studies).  These  steps 
are: 

1.  Establish  an  integrated  solutions  team 

2.  Describe  the  problem  that  needs  solving 

3.  Examine  private  sector  and  public  sector  solutions 

4.  Develop  a  perfonnance  work  statement  (PWS)  or  statement  of  objectives  (SOO) 

5.  Decide  how  to  measure  and  manage  performance 

6.  Select  the  right  contractor 

7.  Manage  performance 

The  intent  is  to  make  the  subject  of  performance-based  acquisition  accessible  for 
all  and  shift  the  paradigm  from  traditional  “acquisition  think”  into  one  of  collaborative, 
performance-oriented  teamwork  with  a  focus  on  program  performance,  improvement, 
and  innovation.  Performance-based  acquisition  offers  the  potential  to  dramatically 
transform  the  nature  of  service  delivery,  and  permit  the  federal  government  to  tap  the 
enormous  creative  energy  and  innovative  nature  of  private  industry. 

Essential  Elements  of  Performance  Based  Acquisition 

In  both  public  and  private  sectors,  performance-based  acquisition  is  viewed  as  a 
strategic  method  to  manage  business  by  promoting  flexibility,  innovation,  and  creating 
win/win  solutions  via  improved  communications,  organizational  goal  alignment,  and 
clear  accountability,  between  the  buyer,  seller(s),  and  subcontractors  (Garrett,  2005). 
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Acquisition  professionals  in  both  industry  and  government  accept  perfonnance-based 
acquisition  as  a  desired  method  in  order  to  increase  perfonnance  by  promoting  flexibility 
through  effective  communication.  According  to  experts  in  US  federal  government 
contracting  and  acquisition  management,  the  essential  elements  of  PB A  are:  performance 
requirements,  perfonnance  standards,  performance  measures  and  metrics,  contractual 
incentives,  and  appropriately  tailored  terms  and  conditions  that  are  specific  to  the  unique 
acquisition  environment  (Garrett,  2005). 

Performance-based  acquisition  is  sometimes  referred  to  as  perfonnance-based 
services  acquisition,  performance-based  contracting,  or  performance-based  services 
contracting.  As  previously  stated,  performance-based  contracting  is  an  acquisition 
method  that  structures  all  aspects  of  an  acquisition  around  the  purposes  of  the  work  to  be 
perfonned  as  opposed  to  either  describing  the  manner  by  which  the  work  to  be 
perfonned,  or  with  broad  or  imprecise  statements  of  work  (Straight,  2006).  Ronald  L. 
Straight  explains  the  primary  essential  elements  of  performance-based  contracting  as: 

■  Describing  the  requirements  in  terms  of  results  rather  than  methods  of 


performance, 

■  Using  measurable  performance  standards  (e.g.,  terms  of  quality, 
timeliness,  etc.)  and  quality  assurance  surveillance  plans, 

■  Including  performance  incentives  where  appropriate  (positive  and 
negative). 


■  The  Federal  Acquisition  Regulation  also  cites  four  elements  of 
Performance  Based  S  ervices  Acquisition: 

(1)  Describing  results  in  terms  of  results  rather  than  methods  of  performance 
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(2)  Using  measurable  performance  standards  and  quality  assurance  plans 

(3)  Specifying  procedures  for  price  or  fee  reductions  when  requirements  are 

not  met 

(4)  Including  performance  appropriate  incentives  where  necessary 
Finally,  Garrett’s  text  defines  clear  and  simple  elements  for  perfonnance-based 

acquisition.  .Garrett  presents  these  elements  as  essential  to  success  (Figure  1). 


Figure  1.  Model  of  Essential  Elements  of  PBA  (Garrett,  2005) 


The  first  element  of  Garrett’s  model  is  people,  and  he  emphasizes  that  the  quality 
of  people  is  much  more  important  than  the  quantity.  Obviously,  the  quality  of  an 
acquisition  team  is  important,  but  if  there  are  not  enough  people  to  execute  the 
acquisition,  the  program  might  fail.  Therefore,  there  is  always  trade-off  between  the 
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quality  of  the  team  and  the  quantity  of  the  team.  Garrett’s  second  element  is  processes, 
as  many  acquisitions  are  complex,  challenging,  and  need  to  be  well  organized.  In  a 
complex  acquisition  environment,  it  is  critical  to  have  and  develop  well-defined, 
documented,  communicated,  and  effectively  implemented  processes  for  leveraging 
competitive  advantages,  managing  opportunities  and  risks,  and  integrating  various 
functional  areas.  Using  enterprise  software  applications,  many  companies  facilitate  more 
disciplined  and  integrated  use  of  business  processes.  The  third  essential  element  of 
performance-based  acquisition  is  performance.  Organizations  use  various  perfonnance 
report  cards  or  balanced  scorecards  to  evaluate  or  assess  the  ability  of  their  organization, 
their  suppliers  so  they  can  meet  or  exceed  their  diverse  acquisition-related  goals.  In  a 
performance-based  acquisition,  defining  the  performance  metrics  is  critical  for  the 
success  of  the  acquisition  program.  The  fourth  and  final  essential  element  of 
performance-based  acquisition  according  to  Garrett  is  price.  In  an  acquisition  program, 
the  right  people  will  develop  the  right  processes,  those  elements  will  drive  the  right 
performance,  and  performance  will  drive  the  right  pricing  strategy,  methods,  and 
arrangements.  The  pricing  strategy  is  usually  detennined  according  to  the  perfonnance 
measures  and  market  related  factors. 

The  Performance-based  management  process/handbook  (1997)  presents  another 
model  of  successful  perfonnance-based  acquisition  programs  consisting  of  six  disciplines 
as  shown  in  Figure  2. 


17 


Figure  2.  Performance-Based  Management  Process/Handbook  Model  (1997) 


Applying  these  six  disciplines  of  performance-based  management  to  contract 
management  helps  drive  performance  and  results  throughout  an  agency’s  culture  and 
business  operations  and  enhances  the  achievement  of  mission  results  (Reed  &Carter, 
2004).  Specifically,  the  steps  outlined  in  the  handbook  are  as  follows: 

1 .  Cultural  Transfonnation:  Proactively  manage  the  organizational  and  cultural 
changes  integral  to  the  success  of  the  initiative, 

2.  Strategic  Linkage:  Provide  a  consistent  vision  throughout  the  organization, 
making  sure  the  desired  results  reflect  organizational  strategic  goals, 

3.  Governance:  Establish  roles,  responsibilities,  and  decision-making  authorities 
for  project  implementation, 

4.  Communications:  Identify  the  content,  medium,  and  frequency  of  information 
flow  to  all  stakeholders; 
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5.  Risk  Management:  Identify,  assess,  monitor,  and  manage  risks;  and 

6.  Performance  Monitoring:  Analyze  and  report  status— cost,  schedule,  and 
performance— on  a  regularly  scheduled  basis  during  project  execution  (PBM  Handbook, 
2001). 

The  National  Perfonnance  Review  (NPR)  Performance  Measurement  Study  Team 
built  a  Performance  Measurement  Process  Model  that  was  published  in  a  June  1997 
report.  This  report  provided  a  useful  frame  of  reference  for  perfonnance  measurement  in 
government  organizations.  The  intergovernmental  benchmarking  team  was  comprised  of 
representatives  from  14  U.S.  federal  agencies,  six  Canadian  government  agencies,  the 
United  Kingdom,  and  two  local  governments  in  the  United  States.  The  team  worked  with 
32  study  partners  drawn  from  more  than  100  organizations  considered  best-in-class  in  the 
area  of  performance  measurement  (NPR  Report,  1997).  Based  on  the  survey  results,  the 
NPR  report  recommends  that  organizations  should  measure  perfonnance  in  the  following 
ways: 

■  setting  goals  and  standards, 

■  detecting  and  correcting  problems, 

■  managing,  describing,  and  improving  processes, 

■  documenting  accomplishments  (NPR  Report,  1997) 

Additionally,  the  report  analyzed  perfonnance  management  and  various  public 
agencies’  management  processes.  The  report  also  developed  a  perfonnance  measurement 
model  that  explains  the  steps,  phases,  and  considerations  involved  in  performance 
measurement.  The  model,  presented  in  Figure  3,  illustrates  the  basic  stages  and  flow  of 
the  perfonnance  measurement  process. 
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Performance  Measurement  Process  Model 
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•  Decision  issues  and 
decision  processes  are 
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1  Data  are  integrated 

1  Analytical  capabilities 
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Evaluating  and 
Utilizing 
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owners  use 
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continuous 
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1  Results  are  displayed 
and  shared  with 
customers  and 
stakeholders 

1  Rewards  and 
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based  on  results 
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comparative 
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class  are  done 

1  Management 
feedback  is  provided 
for  updating  goals 
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Performance 
information  is  used 
to  identify 
opportunities  for 
reengineering  and 
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Figure  3.  NPR  Performance  Measurement  Process  Model  (1997) 


Importance  of  Performance  Based  Acquisition 


Performance-based  practices  are  expected  to  help  the  Department  of  Defense 
(DoD)  improve  performance,  encourage  innovation,  and  increase  competition  in  services, 
often  at  a  reduced  cost  to  the  government  (Gansler,  2000).  The  Air  Force  purchases  a 
broad  range  of  services  to  support  its  installations,  military  and  civilian  employees,  and 
primary  war  fighting  capabilities.  Initial  perfonnance-based  services  acquisition 
implementation  efforts  focused  on  installation  support  services  purchased  through 
operational  contracting  activities  (RAND,  2003). 

The  original  PB  A  performance  goals  were  reviewed  and  a  new  goal  was  set  by  the 
Office  of  Management  and  Budget  to  encourage  the  acquisition  workforce,  including 
program  and  project  managers,  to  employ  PBA  methods  on  service  acquisitions.  The 
memorandum  was  signed  by  Paul  A.  Denett,  Administrator  of  Office  of  Management  and 
Budget  from  the  Executive  Office  of  the  President.  In  fiscal  year  FY2006,  agencies  were 
required  to  apply  PBA  methods  on  40  percent  or  more  of  their  eligible  service  actions 
over  $25,000,  to  include  contracts,  task  orders,  modifications,  and  options,  as  measured 
in  dollars.  According  to  the  Federal  Procurement  Data  System  (FPDS),  most  agencies 
met  or  exceeded  the  goal,  awarding  over  45  percent  of  their  eligible  service  contract 
actions  as  PBAs  (Denett,  2007). 

Recently  a  RAND  (Project  Air  Force)  study  conducted  interviews  with  numerous 
groups  within  the  Air  Force  that  are  using  PBSA  contracts.  They  found  three  key  areas 
related  to  success.  First,  teamwork  is  a  critical  factor  for  success.  Teamwork  encourages 
buy  in,  and  with  any  new  initiative,  the  acquisition  team  must  have  the  support  of  all 
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players  in  the  game.  Second,  market  research  is  important  to  simply  finding  out  what 
resources,  companies,  and  so  on  are  available  to  meet  requirements.  If  properly 
conducted,  market  research  assists  in  applying  commercial  standards  to  requirements 
documents  (for  example,  statements  of  work).  Third,  using  past  performance  information 
in  evaluating  offerors  greatly  enhances  the  chances  of  reaching  a  true  best-value  decision. 
In  addition,  once  contractors  realize  past  performance  is  being  evaluated,  they  are  more 
apt  to  perform  better  to  get  more  business  in  the  future  (RAND,  2001). 

The  Project  Air  Force  study  indicates  that  those  who  were  interviewed  were  very 
happy  with  the  perfonnance  of  the  vendors  they  had  obtained  using  PBSA  practices. 
Determining  the  effects  on  the  cost  was  very  difficult  because  of  difficulty  of  detennining 
internal  costs  and  accuracy  of  govermnent  cost  estimates.  This  study  also  indicates  that 
the  customers,  contracting  officers,  and  vendors  were  generally  satisfied  and  happy  with 
PBSA  approaches  to  contracting. 

Another  RAND  study,  Acquisition  Reform  Cost  Savings  Estimates,  indicates  that 
requirements  reform  (performance-based  specifications)  and  cost  as  an  independent 
variable  (CAIV)  are  crucial  for  cost  savings.  Although  there  were  wide  variations  from 
one  system  or  program  to  another,  the  RAND  report  found  that  some  programs  (such  as 
JDAM  and  various  avionics  efforts)  could  claim  large  savings  from  acquisition  reform. 
Although  the  primary  focus  on  the  multiyear  contract  cost  savings,  rather  than 
performance-based  acquisitions,  the  findings  are  nevertheless  relevant  from  the 
performance-based  acquisition  perspective.  The  study  points  out  some  advantages  of 
multi-year  contracting.  Some  benefits  of  multi-year  contracting  related  to  PBA  are:  the 
avoidance  of  nonrecurring  costs  associated  with  negotiating  and  implementing  a  new 
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contract  each  year,  long-tenn  hiring  and  personnel  planning  can  be  made  more  efficient, 
and  the  contractor  can  plan  the  purchase  of  long  lead  items  and  materials  longer 
production  runs  (RAND  study,  2001).  Estimates  for  multiyear  contracting  savings  for  a 
selected  list  of  programs  are  illustrated  in  Table  3. 


Table  3.  An  Overview  of  Acquisition  Reform  Cost  Savings  Estimates 


Program  b 

Production  Savings  (%) 

Estimate  Quality 

F-16  (FY82-85) 

10 

Forecast 

F-16  (FY86-89) 

10 

Forecast 

F-16  (FY90— 93) 

5.5 

Forecast 

F-16  (FY99-02) 

5.4 

Forecast 

CDE  for  C-17 

8.2 

Forecast 

C-17  (airframe) 

5.5 

Forecast 

Javelin  ATGM 

14.3 

Analysis 

MTVR 

7.4 

Analysis 

CH-60  (U.S.  Navy  and  U.S.  Army) 

5.5 

Forecast 

DDG-51  (FY98-01) 

9 

Forecast 

F-22  (1996  CA1G/JET) 

3.9— 4.7 

Analysis 

F/A-18E/F  (target) 

7.4 

Analysis 

aSavings  percentages  include  government  investments  for  cost  reduction  inidatives 
for  C-17  airframe  and  F/A-18E/F. 

^CDE  =  Commercial  Derivative  Engine;  ATGM  =  anti-tank  guided  missile;  MTVR  = 

Medium  TacUal  Vehicle  Replacement;  CAIG  =  Cost  Analysis  Improvement  Group;  JET 
=  Joint  Estimate  Team. 

(Source:  RAND  study,  2001) 


Since  the  end  of  the  Cold  War,  base  closure  and  realignment  (BRAC)  actions 
have  reduced  the  number  of  military  depots  in  the  United  States  from  38  to  19  and 
manpower  by  two-thirds  (Mahon,  2007).  In  parallel  with  the  DoD’s  shrinking  logistics 
support  base,  the  fleets  have  continued  to  age,  weapon  system  and  parts  availability  have 
continued  to  decline,  obsolescence  has  increased,  and  vendors  have  disappeared,  leading 
to  increased  DoD  sustainment  costs.  With  these  facts,  the  desire  to  better  manage  the 
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total  life-cycle  support  costs  of  weapons  systems  and  improve  logistics  support  to  the  war 
fighter,  resulted  in  the  selection  of  Performance  Based  Logistics  (PBL)  as  a  key  logistics 
transfonnation  initiative  (Mahon,  2007). 

Since  the  performance-based  environment  surrounding  DoD  weapons  systems  is 
so  complex  (e.g.  C-17),  availability,  reliability,  and  cycle  time  are  typically  measured. 
Frequently,  there  are  no  metrics  to  measure  cost-savings  or  reduced  logistics  footprint.  In 
some  cases  (C-17),  there  may  not  be  convincing  evidence  that  PBL  is  saving  the  air  force 
anything  on  these  objectives  (Mahon,  2007). 

An  acquisition  can  be  separated  into  two  major  milestones:  system  acquisition 
and  system  sustainment.  The  government’s  primary  concern  has  been  on  system 
acquisition.  However,  the  sustainment  costs  for  a  system  is  often  much  higher  than  the 
acquisition  of  the  system  itself.  It  has  been  estimated  that  about  30  percent  of  all  dollars 
spent  are  used  to  acquire  the  system,  while  the  remaining  70  percent  of  all  dollars  are 
used  for  support  (Berkowitz,  et.al.,  2005).  In  their  study,  Berkowitz  and  his  colleagues 
interviewed  managers  from  30  different  initial  DoD  acquisition  programs.  The  study 
indicates  that  in  order  to  meet  the  PBL  objectives,  both  government  and  industry  must 
agree  on  business  practices  that  provide  the  greatest  value  for  all  parties  (Berkowitz,  et 
al„  2005). 

Two  case  studies,  a  contract  to  Lockheed  Martin  in  support  of  the  S-3  aircraft,  and 
a  repair  contract  to  Pratt  and  Whitney  in  support  of  the  J52  engine,  clearly  explain  the 
advantages  of  PBL  contracts.  Some  of  these  advantages  include  the  elimination  of  dual 
infrastructures,  reductions  in  materiel  costs,  increased  government  accountability, 
enhanced  commercial  partnerships,  increased  parts  availability,  increased  focus  by  the 
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PBL  vendor,  and  benefits  to  small  businesses.  These  advantages  have  enhanced  support 
to  the  J52,  S-3  and  other  PBLs  where  DLA  has  partnered  with  private  companies  (Starks, 
2005). 

In  2005,  Defense  Acquisition  University  (DAU)  published  Performance-Based 
Logistics:  A  Program  Manager  s  Product  Support  Guide.  This  guide  is  comprised  of 
success  stories  about  the  implementation  of  PBL.  F-117,  JSTARS,  and  TOW  ITAS  are 
Reduction  in  Total  Cost  (RTOC)  pilot  programs  that  benefited  from  the  PBL  efforts. 
Those  programs  have  been  highly  successful,  reaping  significant  cost  savings/avoidance 
and  identifying  lessons  learned  which  are  now  being  institutionalized  by  DoD  (DAU, 
2005). 

Cultural  Effects  on  Acquisition  Environment 

In  April  2000,  the  Under  Secretary  of  Defense  Dr.  Gansler  established  a  goal  for 
the  U.S.  Department  of  Defense  to  have  the  minimum  50%  of  acquisitions  procured  by 
the  perfonnance-based  acquisition  method  by  2005.  By  2007,  DoD  had  more  than  100 
acquisition  programs  supported  by  the  PBA,  the  Joint  Strike  Fighter  (F-35)  being  one  of 
six  originally  selected  pilot  programs.  Current  expectations  are  that  the  F-35’s  life-cycle 
cost  will  be  about  20%  lower  when  compared  to  that  of  legacy  systems  such  as  F/A-16 
and  F/A-18  (Sols,  Nowick,  Verma,  2007). 

Although  the  PBA  is  a  new  method  in  the  acquisitions  area,  many  countries  and 
private  sectors  have  implemented  this  procedure  and  started  to  use  it  in  their  logistics 
systems.  The  United  Kingdom’s  (U.K.)  Defense  Logistics  Organization  is  transitioning 
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to  performance-based  contracting  support  for  availability  and  capability  for  the  sake  of 
seeking  achievement  of  established  effectiveness  levels.  The  U.K.’s  Royal  Air  Force  is 
expecting  to  reduce  the  costs  by  12%  in  the  outsourced  maintenance  of  E-3D  Sentry 
Airborne  Warning  and  Control  System  (AWACS)  aircraft  (Sols,  Nowick,  Venna,  2007). 
European  Aeronautic  Defense  and  Space  Company  Construcciones  Aeronauticas  S.A. 
(EADS  CASA)  has  a  defense  acquisition  relationship  with  many  South  American 
countries  including  Chile,  Ecuador,  Paraguay,  and  Brazil.  Those  countries  entered  into 
the  performance-based  acquisition  area  with  the  purchase  of  the  C-212  and  CN-235 
aircraft  systems. 

The  U.S.  Department  of  Defense,  many  South  American  countries,  and  some 
European  countries  have  been  applying  PBA  in  their  acquisition  systems.  Many  other 
countries  are  acquiring  the  same  weapon  systems  through  joint  venture  acquisitions. 
Those  countries  have  been  using  performance-based  acquisition  methods,  at  least 
indirectly.  It  is  important  to  learn  whether  cultural  differences  play  a  big  role  in  this  the 
success  or  failure  of  performance-based  acquisition  methods. 

According  to  a  study  conducted  by  Simcha  Ronen  and  Oded  Shenkar,  there  are 
eight  culture  clusters  in  the  world.  Countries  and  cultures  are  “clustered”  according  to 
similarities  on  certain  cultural  dimensions  such  as  work  goals,  values,  needs,  and  job 
attitudes  (Ronen,  S.  Shenkar,  O.,  1983).  As  the  acquisition  environment  becomes  more 
complex,  multinational  companies  increase  their  direct  investment  overseas,  especially  in 
less  developed  and  consequently  less  studied  areas,  they  will  require  more  information 
concerning  their  local  employees  in  order  to  implement  effective  types  of  interactions 
between  the  organization  and  the  host  country  (Ronen,  S.  Shenkar,  O.,  1983).  The  study 
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produced  helpful  knowledge  to  understand  better  the  work  values  and  attitudes  of 
employees  throughout  the  world.  The  authors  claim  that  American  theories  work  very 
well  for  Western  nations  and  they  keep  the  study  open  for  future  research  by  asking,  “are 
those  theories  applicable  for  non-Westem  countries?”  A  synthesis  of  country  clusters  is 
shown  in  Figure  4.  The  countries,  United  States,  United  Kingdom,  Canada,  Australia, 
and  Ireland  were  characterized  as  former  British  colonies  and  are  included  in  Anglo 
cluster.  The  Near  Eastern  cluster  contains  Greece,  Iran,  and  Turkey.  This  cluster  is 
characterized  by  high  power  distance,  high  uncertainty  avoidance,  low  individualism,  and 
medium  masculinity. 


Figure  4.  A  synthesis  of  Country  Clusters  (Ronen  and  Shenkar,  1985). 
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In  another  study,  Chang  (2004)  emphasized  the  regulation  of  foreign  investment 
from  a  historical  perspective.  More  advanced  European  economies  such  as  UK,  France, 
and  Germany  have  used  fonnal  or  informal  performance  requirements  for  key  Foreign 
Direct  Investment  (FDI)  projects  to  ensure  that  their  national  interests  are  not  hurt  in 
sensitive  sectors  such  as  defense  and  cultural  industries.  East  Asian  countries  such  as 
Japan,  Korea,  and  Taiwan  have  also  applied  very  liberal  policies  in  certain  sectors  while 
being  very  restrictive  in  others  (Chang,  2004).  Because  perfonnance-based  acquisition  is 
a  new  concept  and  includes  more  interactions  between  government  and  contractor  firms, 
there  is  likely  to  be  resistance  to  change.  This  resistance  can  be  overcome  through 
effective  training  of  acquisition  managers  and  filling  the  gaps  with  regulations  that  will 
benefit  both  government  and  vendors. 

Human  Resource  Management  (HRM)  policies  can  be  significant  control 
mechanisms  for  companies  and  governments  engaged  in  cross-cultural  acquisitions 
(Faulkner,  et  al.,  2002).  Faulkner  and  his  colleagues  conducted  a  study  that  examines 
whether  there  are  such  things  as  best  HRM  practices  consistent  across  nationalities  or 
whether  companies  of  different  nationalities  generally  adopt  HRM  strategies.  Table  4 
shows  the  comparison  of  national  differences  in  HRM.  The  study  shows  substantial 
convergence  in  certain  practices  (perfonnance-related  pay,  increased  training,  and  team- 
based  product  development),  but  there  remained  considerable  national  differences  in 
HRM  practices. 
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Table  4.  Comparison  of  National  HRM  practices  (Faulkner,  et.al,  2002) 


US 

Japan 

Germany 

France 

UK 

Pay 

Performance- 

related 

PRP  growing 

PRP  growing 

PRP  growing 

PRP  growing 

Recruitment 

Short  term 

Lifetime 

Long  term 

French  long; 
local  less  so 

Less  short 

term  than  US 

Training 

High; 

courses 

On-the-jo  b 

Technical  bias 

To  a  ceiling 

Increased ; 

courses 

Career 

planning 

Little 

Steady  and 
slow 

Ad  hoc 

Highly 

structured 

Very  variable 

Product 

development 

Strong  move 

towards 

team-based 

Fairly  strong 
team -based 

Strong  move 
to  team- based 

Some  use 

of  teams 

Not  very 
team -based 

Culture 

Top-down 

Bottom-up: 

consensual 

Top-down 

Top-down 

Top-down 

Appraisal 

Regular  and 
formal 

Subtle;  not 
transparent 

Growing 

Growing 

Regular  annual 

Promotion 

Fast  and 

performance- 

based 

Slow  and 
seniority- 
based 

Based  on 

technical 

expertise 

Emphasis  on 
formal 

qualification; 

fast 

Variable 

Communications 

Formal: 

need-to-know 

Open  when 
asked 

Open  and 
Informal 

Open  and 
formal 

Need-to-know 

approach 

(Note:  PRP  stands  for  Performance-Related  Pay). 


Summary 

This  chapter  described  the  literature  relevant  to  the  creation,  definition,  examples, 
and  evaluation  of  the  performance-based  acquisition.  The  chapter  differentiated  the 
essential  elements  defined  by  Straight,  the  FAR,  and  Garrett  and  explained  the  primary 
essential  elements  to  the  success  of  performance-based  acquisitions.  This  chapter  also 
addressed  why  organizations  may  benefit  from  building  a  perfonnance-based 
environment  for  acquisitions.  Results  of  some  success  stories  in  the  performance-based 
logistics  environment  were  shown  to  emphasize  the  importance  of  PBA.  Finally,  this 
chapter  addressed  cultural  differences  in  literature  related  to  the  acquisition  environment. 
The  following  chapter  will  discuss  the  methods  used  to  conduct  the  research. 
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III.  Methodology 


Chapter  Overview 

The  purpose  of  this  chapter  is  to  describe  the  chosen  methodology  for  organizing 
the  research  design  and  collecting  relevant  data.  In  order  to  organize  the  study,  the 
researcher  reviewed  Yin  (2003),  Locke  (2001),  Glaser  et  al  (1967)  and  Maxwell  (2005), 
and  selected  a  qualitative  approach  with  grounded  theory  as  the  specific  tradition  of 
inquiry.  Using  guidance  from  qualitative  research  literature,  a  semi-structured 
questionnaire  was  prepared  which  served  as  the  data  collection  tool  for  the  target 
question  of  the  study.  This  set  of  questions  was  designed  to  obtain  infonnation  from 
experienced  acquisition  professionals  regarding  perfonnance-based  acquisition  methods. 
The  researcher  transcribed,  categorized,  and  coded  data  from  the  interviews  into  a 
Microsoft  Excel  spreadsheet  in  accordance  with  qualitative  methodological  precepts. 

Research  Design 

Research  can  serve  many  purposes.  Earl  Babbie  mentions  the  most  common 
three  purposes  of  research  in  his  “The  Basics  of  Social  Research”  book.  Those  purposes 
are  exploration,  description,  and  explanation.  Much  of  social  research  is  conducted  to 
explore  a  topic  and  this  approach  typically  occurs  when  a  researcher  examines  a  new 
interest  or  when  the  subject  itself  is  relatively  new  (Babbie,  2005).  As  the  topic  of  this 
study  is  relatively  new  (i.e.  performance-based  acquisition  in  an  international  context), 
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the  researcher  chose  an  exploratory  research  approach.  Exploratory  studies  are  typically 
done  for  three  purposes:  (1)  to  satisfy  the  researcher’s  curiosity  and  desire  for  better 
understanding,  (2)  to  test  the  feasibility  of  undertaking  a  more  extensive  study,  and  (3)  to 
develop  the  methods  to  be  employed  in  any  subsequent  study  (Babbie,  2005). 
Exploratory  studies  can  also  be  a  source  for  grounded  theory  (Babbie,  2005),  providing 
new  models  for  future  explanatory  research. 


Qualitative  Research 


According  to  Creswell,  there  are  three  distinct  approaches  to  research  design: 
quantitative,  qualitative,  and  mixed  method  (Creswell,  2003:17).  Creswell  defines 
quantitative  research  as: 

...One  in  which  the  investigator  primarily  uses  post  positivist  claims  for 
developing  knowledge  (i.e.  cause  and  effect  thinking,  reduction  to  specific 
variables  and  hypotheses  and  questions,  use  of  measurement  and 
observation,  and  the  test  of  theories),  employs  strategies  of  inquiry  such  as 
experiments  and  surveys,  and  collects  data  on  predetennined  instruments 
that  yield  statistical  data  (Creswell,  2003:18). 

In  contrast  to  quantitative  research,  he  defines  qualitative  research  as: 

. .  .One  in  which  the  inquirer  often  makes  knowledge  claims  based 
primarily  on  constructivist  perspectives  or  advocacy/participatory 
perspectives  or  both.  It  also  uses  strategies  of  inquiry  such  as  narratives, 
phenomenologies,  ethnographies,  grounded  theory  studies,  or  case  studies. 

The  researcher  collects  open  ended,  emerging  data  with  the  primary  intent 
of  developing  themes  from  the  data  (Creswell,  2003:18). 

To  select  an  appropriate  research  approach  for  a  given  study,  the  researcher 

should  be  able  to  define  the  relationship  between  the  problem  and  a  particular  research 

approach.  As  an  example,  a  researcher  seeking  to  identify  or  explain  the  relationship 
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between  measured  variables  (e.g.  numerical  quantities)  would  likely  be  best  suited  to 
select  a  quantitative  approach  (Leedy  and  Ormrod,  2001).  In  contrast,  if  the  researcher’s 
aim  were  to  explore  and  characterize  existing  phenomena,  or  to  develop  a  theory 
regarding  such  phenomena,  a  qualitative  method  would  likely  be  the  better  choice  (Leedy 
and  Ormrod,  2001).  In  summary,  Table  5  identifies  the  three  types  of  approaches 
available  to  researchers. 


Table  5.  Research  Method  Procedures  (Creswell,  2003:17) 


Quantitative 

Qualitative 

Mixed  Methods 

■  Predetermined 

•  Emerging  methods  ^ 

-Both predetermined  and 

-  Instmment  based  questions  > 

*<ipen-ended  questions 

emerging  methods 

c  c 

■  Performance  data,  attitude^ 

■  Interviewdata,  observation 

)- Both  open- and  closed- 

data,  observational  data,  ' 

v^at^dociunent  datajnj^ 

ended  questions 

and  census  data 

auSovisimi  dlltcl 

-Multiple  forms  of  data 

-  Statistical  analysis 

■  Text  and  image  analysis 

drawing  on  all  possibilities 
-Statistical  and  text  analysis 

In  choosing  the  appropriate  research  method  for  a  particular  study,  perhaps  the 
most  important  factor  is  the  nature  of  the  data  itself.  Data  and  methodology  are  always 
interdependent.  The  methodology  to  be  used  for  a  particular  research  problem  must 
always  take  into  account  the  nature  of  the  data  (Leedy  and  Ormrod,  2001). 

The  researcher  chose  a  qualitative  research  method  that  was  appropriate  for  data 
collection  in  this  study.  After  deciding  to  follow  a  qualitative  methodology,  the  specific 
tradition  of  inquiry  to  guide  the  research  needed  to  be  determined.  Although  there  are 
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many  other  accepted  traditions,  Creswell  identified  five  popularly  accepted  traditions 
frequently  used  (Creswell,  1998:5).  The  five  traditions  are  biography,  phenomenological 
study,  grounded  theory  study,  ethnography,  and  case  study. 

The  approach  used  for  qualitative  data  collection  method  for  this  study  was  a 
semi- structured  interview  with  subject  matter  experts.  The  interview  questions  attempted 
to  capture  the  experiences  of  acquisition  professionals  within  actual  programs  that 
interviewees’  had  been  working  on.  The  nature  of  the  semi-structured  interview 
questions  was  similar  to  a  multiple  case  study.  Thus,  the  study  will  be  a  hybrid  study, 
using  both  grounded  theory  and  case  study  methods,  and  the  following  sections  address 
these  two  topics,  respectively. 

Grounded  Theory  Approach 

Grounded  Theory  emerged  from  the  collaboration  of  two  sociologists:  Barney 
Glaser  and  Anselm  Strauss.  They  built  this  theory  from  combining  two  main  traditions 
of  research:  positivism  and  interactionism.  Grounded  Theory  can  be  described  as  an 
approach  that  attempts  to  combine  a  naturalist  approach  with  a  positivist  concern  for  a 
“systematic  set  of  procedures”  in  doing  qualitative  research  (Babbie,  2005).  Creswell 
defines  grounded  theory  as  a  study  to  generate  or  to  discover  a  theory,  an  abstract 
analytical  schema  of  a  phenomenon  that  relates  to  a  particular  situation.  This  situation  is 
one  in  which  individuals  interact,  take  action,  or  engage  in  a  process  in  response  to  a 
phenomenon  (Creswell,  1998). 
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According  to  Glaser  and  Strauss,  a  theory  is  a  strategy  for  handling  data  in 
research,  providing  modes  of  conceptualization  for  describing  and  explaining.  “The  form 
in  which  a  theory  is  presented  does  not  make  it  a  theory;  it  is  a  theory  because  it  explains 
or  predicts  something”  (Glaser  &  Strauss,  1967).  Grounded  theory  can  be  presented 
either  as  a  well  codified  set  of  propositions  or  in  a  running  theoretical  discussion,  using 
conceptual  categories  and  their  properties.  While  Glaser  and  Strauss  recognize  that 
qualitative  and  quantitative  data  are  both  useful  for  verification  and  generation  of  theory, 
their  grounded  theory  approach  focuses  on  qualitative  data.  They  explain  that  qualitative 
methods  work  best  in  discovering  the  crucial  elements  of  sociological  theory,  that  is, 
these  crucial  elements  are  best  derived  from  data  on  structural  conditions,  consequences, 
deviances,  norms,  processes,  patterns,  and  systems.  The  grounded  theory  approach  is 
most  advantageous  in  difficult  empirical  situations,  such  as  new  or  unusual  environments 
(Glaser  &  Strauss,  1967). 

The  Discovery  of  Grounded  Theory  explains  that  researchers  assign  meaning  to 
their  observations  in  terms  of  the  constant  comparative  method.  This  set  of  practices 
offers  logic  for  composing  conceptual  elements  that  hinge  on  their  articulation  through 
close  reading,  comparison  and  attendant  conceptualization  of  data  (Locke,  2003).  Locke 
(2003)  conceptualizes  the  constant  comparative  method  in  terms  of  four  stages,  which 
span  the  entire  study,  moving  from  assigning  meaning  to  incidents  of  recorded  data  to 
refining  and  writing  up  the  completed  theoretical  framework. 

Stage  1:  comparing  incidents  applicable  to  each  category 

When  researchers  engage  their  data  during  the  forms  of  the  constant  comparative 
method,  they  participate  in  two  activities,  naming  data  incidents  and  comparing  data 


34 


incidents  and  names.  These  two  activities  are  complemented  by  a  third,  supportive 
activity,  memoing.  In  naming,  researchers  conceptualize  and  develop  the  abstract 
meaning  of  incidents  by  articulating  what  they  perceive  is  being  expressed  in  those 
incidents.  Comparing  occurs  in  tandem  with  naming  and  is  critical  to  help  researchers  to 
develop  a  category  for  multiple  observations  and  it  helps  to  clarify  what  the  researcher 
perceives  from  the  data.  As  the  researcher  begins  to  identify  the  properties  of  a  category 
and  to  be  caught  in  various  ideas  that  the  comparative  process  engenders,  recording  a 
memo  on  the  emerging  ideas  captures  these  fresh  theoretical  musings  and  gives  the 
researcher  analytic  space  to  reflect  and  work  out  these  ideas  (Locke,  2003). 

Stage  2:  integrating  categories  and  their  properties 

In  this  stage,  researcher  aims  to  develop  an  organization  for  the  conceptual 
categories  derived  in  stage  1 .  In  order  to  arrange  those  categories,  the  researcher  is 
concerned  with  how  the  various  conceptual  elements  can  be  arranged  in  relation  to  each 
other.  At  this  stage  of  analytic  activity,  writing  memos  can  help  the  researcher  to 
articulate  the  significance  of  the  categories  and  the  relationships  between  the  analytic 
elements  in  the  framework. 

Stage  3:  delimiting  the  theory 

This  stage  sets  boundaries  around  the  theory.  Here  the  aim  is  to  settle  on  the 
theoretical  components  and  to  clarify  the  story  they  have  to  tell  about  the  phenomenon  or 
social  situation.  Once  the  researchers  have  integrated  their  theoretical  categories  and 
make  a  commitment  to  tell  a  particular  story  from  the  data,  this  process  of  boundary¬ 
setting  will  help  the  researchers  to  focus  on  the  more  relevant  and  robust  categories. 
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Stage  4:  writing  the  theory 

In  this  final  stage,  the  researcher  is  able  to  produce  a  research  or  story  by  virtue  of 
possessing  coded  data,  a  series  of  memos,  and  a  theory.  Locke’s  four  stages  of  the  theory 
development  process  are  shown  in  Figure  5. 


Integrating 

Delimiting 

* 

Writing 

Figure  5.  Four  stages  of  theory  development  process  (Locke,  2003). 


Case  Study  Methodology 

Since  interviewees  responded  to  questions  from  their  experiences  in  different 
acquisition  contexts  (different  programs,  different  countries,  etc.),  this  study  amounts  to  a 
multiple  case  study.  In  determining  the  appropriate  case  study  strategy  there  are  three 
conditions  that  apply: 

1)  the  types  of  research  questions, 

2)  the  extent  of  control  an  investigator  has  over  actual  behavioral  events,  and, 

3)  the  degree  of  focus  on  contemporary,  as  opposed  to  historical,  events  (Yin,  2003). 
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The  types  of  research  questions  is  the  familiar  series  of  “who,”  “what,”  “where,” 
“how,”  and  “why.”  If  the  research  questions  focus  on  “what”  questions,  this  type  of 
question  is  a  justifiable  rationale  for  conducting  an  exploratory  study.  When  the  study 
focuses  on  the  “who”  and  “where”  type  of  questions,  the  study  is  likely  to  favor  the 
analysis  of  archival  records,  as  in  economic  research.  In  contrast,  “how”  and  “why” 
questions  are  more  explanatory  and  likely  to  lead  to  case  studies,  histories,  and 
experiments.  Defining  the  research  questions  is  probably  the  most  important  step  in 
research.  The  research  questions  should  have  both  substance  (What  is  the  study  about?) 
and  form  (“who,”  “what,”  “where”  questions)  (Yin,  2003).  Since  the  nature  of  the  study 
is  to  explore  the  success  of  the  PBA  in  different  cultural  environments,  the  semi- 
structured  questionnaire  includes  more  “what”  and  “why”  questions. 

The  researcher  had  little  to  no  control  over  actual  behavioral  events. 

Additionally,  the  researcher  focused  on  contemporary  or  recent  past  events.  Therefore, 
the  study  was  non-experimental,  and  non-historical.  Instead,  the  researcher  used  the  case 
study  method.  “Although  case  studies  and  histories  look  similar,  the  case  study’s  unique 
strength  is  the  ability  to  deal  with  a  full  variety  of  evidence-documents,  artifacts, 
interviews,  and  observations”  (Yin,  2003,  p.8).  The  case  study  relies  on  many  of  the 
same  techniques  as  history,  but  it  adds  direct  observation  of  events  and  interviews  of  the 
persons  involved  in  the  events. 
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Data  Collection 


The  most  commonly  used  data  sources  in  doing  qualitative  research  are 
documentation,  archival  records,  interviews,  direct  observations,  participant  observations, 
and  physical  artifacts  (Yin,  2003).  According  to  Creswell,  the  four  basic  forms  of 
qualitative  data  are:  1)  observations,  2)  interviews,  3)  documents  and  4)  audio-visual 
materials.  Importantly,  he  notes  that  each  form  of  data  has  both  advantages  and 
limitations  that  the  researcher  should  consider  when  planning  the  research  design 
(Creswell,  2003). 

Since  performance-based  acquisition  is  a  relatively  new  method  in  the  acquisition 
area,  selecting  the  data  collection  process  was  an  important  consideration  for  the 
researcher.  Since  many  program  managers  were  available  within  AFIT  and  a  variety  of 
different  countries’  liaison  officers  working  with  FMS  were  located  in  the  Wright- 
Patterson  Air  Force  Base,  conducting  interviews  was  the  best  way  to  obtain  convenient 
qualitative  data.  Yin  indicates  that  one  of  the  most  important  sources  of  case  study 
infonnation  is  the  interview.  The  data  were  collected  using  open-ended  interviews  with  a 
total  of  thirteen  liaison  officers,  program  managers,  contracting  officers,  and  contracting 
negotiators.  In  order  to  prepare  a  well-defined  semi-structured  questionnaire,  the 
researcher  considered  these  two  elements:  (1)  to  follow  the  subject  of  the  study’s  line  of 
inquiry,  and  (2)  to  ask  conversational  questions  in  an  unbiased  manner.  The  semi- 
structured  interview  questions  are  shown  in  Table  6. 
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Table  6.  Semi-Structured  Interview  Questions 


No: 

Question: 

1 

Have  you  ever  been  involved  in  a  contract  using  performance  based  acquisition 
methods?  If  so,  can  you  describe  the  methods  that  were  used?  What 
“worked”?  What  did  not  “work”?  In  your  estimation,  did  perfonnance  based 
methods  result  in  cost,  schedule,  or  performance  benefits  compared  to  other 
methods? 

2 

Have  you  been  involved  with  non-U. S.  contracts?  If  so,  did  they  use 
perfonnance  based  acquisition  methods?  If  so,  could  you  describe  how  those 
methods  were  implemented?  Would  you  say  that  the  methods  were 
implemented  substantially  the  same  as  they  would  be  for  a  U.S.  contract? 
Substantially  different? 

3 

In  the  future,  do  you  plan  to  implement  performance  based  acquisition  methods 
(in  a  U.S.  environment  or  in  a  non-U. S.  environment)?  Do  you  anticipate  any 
advantages?  Disadvantages? 

4 

In  acquisition  programs  in  which  you  have  been  involved,  what  has  been  the 
most  critical  issue  or  obstacle  to  success  (e.g.  cost,  sustainability,  meeting  the 
user’s  requirements,  schedule,  etc.)? 

5 

Can  you  share  an  acquisition  “horror  story”  with  me,  when  something  went 
wrong? 

6 

Can  you  share  an  acquisition  “hero  story”  with  me,  when  something  went 
right? 

7 

When  can  you  describe  an  acquisition  as  successful?  What  are  the  most 
important  metrics  for  determining  success? 

Summary 

This  chapter  described  the  research  design  and  specific  methodology  selected  to 
conduct  the  thesis  study.  Due  to  the  nature  of  the  research  design  and  data,  the  researcher 
selected  a  qualitative  research  method.  In  order  to  obtain  convenient  data,  a  semi- 
structured  interview  questionnaire  was  prepared  including  seven  investigative  questions. 
These  questions  were  asked  to  liaison  officers,  program  managers,  contracting  officers, 
and  contracting  negotiators.  Interviews  were  recorded  with  the  interviewee’s  consent  and 
then  every  sentence  was  transcribed  into  an  Excel  worksheet.  Each  sentence  was  then 
categorized  using  the  coding  and  memoing  process.  Finally,  the  data  were  sorted  into 
categories  to  see  if  patterns  emerged  related  to  specific  research  questions  that  could 


39 


ultimately  lead  to  new  theory  in  the  area  of  performance-based  acquisition  in  an 
international  context.  The  following  chapter  will  discuss  the  analysis  of  data  of  the 
research. 
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IV.  Data  Analysis 


Overview 

The  purpose  of  this  chapter  is  to  document  and  analyze  the  qualitative  data 
gathered  during  the  semi-structured  interview  portion  of  this  research.  As  previously 
discussed,  the  questionnaire  included  seven  open-ended  questions.  The  thirteen 
interviewees  included  program  managers,  contracting  officers,  liaison  officers,  and 
project  managers.  Interviews  were  analyzed  and  transcribed  into  an  Excel  worksheet 
using  coding  theory  (Locke,  2003).  Each  sentence*  was  given  a  code  to  retrieve  and 
organize  the  themes  of  the  interviews.  The  organizing  part  of  the  coding  entails 
systematic  categorization  of  the  various  themes,  so  the  research  can  quickly  find,  pull  out, 
and  cluster  the  segments  relating  to  the  primary  investigative  question  (Miles  & 
Huberman,  1994).  Clustering  is  used  to  draw  conclusions  from  the  themes  that  emerge 
from  the  interviews.  In  order  to  provide  a  more  in-depth  view  into  the  interview 
responses,  a  complete  transcription  of  all  interviews  is  included  as  Appendix  B. 

Demographics 

The  qualitative  data  were  gathered  from  a  total  of  thirteen  program  managers, 
contracting  negotiators,  liaison  officers,  and  project  managers.  The  researcher  chose 


*  The  term  “sentence”  may  refer  to  multiple  sentences  in  this  study.  The  researcher  assigned  to  single  or 
multiple  sentences  a  common  meaning  represented  by  a  single  code  that  is  captured  or  composed  in  a 
conceptual  category. 
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interviewees  from  a  variety  of  acquisition  areas  in  order  to  reduce  possible  biased  results. 
Performance-based  acquisition  is  a  relatively  new  method,  and  interviewees’  experiences 
differ  according  to  the  programs  in  which  they  have  been  involved  and  their  own 
government  acquisition  officer  training  policies.  The  demographics  of  interviewees  are 
illustrated  in  Figure  6. 


■  Chief  of  Test  Project 

■  Contracting  Negotiator 

■  deputy  program  Manager 

■  Planning  Division  Chief  at 
Materiel  Command  Stuff 

■  Liasion  Officer 

■  Project  Manager 


■  Program  Manager 


Figure  6.  Demographics  of  Interviewees 


The  experience  levels  of  the  interviewees  were  collected.  Interviewees,  who  had 
less  than  ten  years  of  experience,  were  typically  young  officers  or  were  new  to  the  PBA 
environment.  On  the  other  hand,  interviewees  who  had  more  than  ten  years  of  experience 
had  worked  in  multiple  acquisition  programs  or  they  were  involved  in  big,  complex 
acquisition  programs.  Most  of  the  interviewees  were  program  managers  and  have  an 
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average  of  four  years  of  experience  in  the  acquisition  area.  Table  7  shows  the  job  title 
and  the  experience  of  interviewees. 


Table  7.  Interviewees’  Job  title  and  Experience 


Job  Title 

Experience 

Chief  of  Test  Project 

5  years 

Contracting  Negotiator 

7  years 

Deputy  Program  Manager 

22  years 

Liaison  Officer 

8  years 

Liaison  Officer 

2  years 

Planning  Division  Chief  at  Materiel  Command  Staff 

3  years 

Program  Manager 

3  years 

Program  Manager 

6  years 

Program  Manager 

3  years 

Program  Manager 

4  years 

Program  Manager 

3  years 

Project  Manager 

3  years 

Project  Manager 

10  years 

Interview  Analysis 

Using  the  grounded  theory  method,  every  sentence  from  every  interview  was 
“coded”  to  gain  insight  and  allow  meaningful  patterns  to  emerge.  The  researcher  coded 
sentences  into  categories  and  subcategories.  In  order  to  reduce  researcher  bias,  the 
researcher  coded  sentences  according  to  the  core  meanings  of  sentences  and  intentions  of 
the  interviewees.  An  example  of  the  open  coding  approach  is  shown  in  Tables  8  and  9. 

In  the  following  examples,  interviewee’s  sentences  were  transcribed  into  an  Excel 
worksheet  and  then  the  researcher  looked  for  the  meaningful  pattern  leading  to  the  core 
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meaning  of  the  sentences.  In  the  first  example,  illustrated  in  Table  8,  the  interviewee 
talks  about  how  unstable  user  needs  create  a  challenge  for  the  acquisition  program. 


Table  8.  A  Coding  Example  for  Challenges 


Q# 

SENTENCE 

SUBCATEGORY 

CATEGORY 

5 

It  was  a  development  of  an  emulator;  it  was  a 
piece  of  test  equipment,  that  would  take  care  of 
some  older  technology  and  we  were  trying  to 
develop  this  emulator.  The  development  of 
emulator  was  in  progress  and  we  had 
requirements.  Unfortunately,  one  of  the 
engineers  unknown  to  the  program 
management  was  working  with  the  contractor 
and  adjusting  the  requirements.  You  had  a 
changing  requirements  base  and  that  resulted  in 
the  contractor  having  to  do  additional  work. 

Unstable 
user  needs 

Challenges 

In  the  second  example,  shown  in  Table  9,  the  interviewee  refers  to  the  factors  that 
are  needed  for  Program  Success.  The  interviewee  describes  the  importance  of  having 
good  communication  with  the  contractor. 


Table  9.  A  Coding  Example  for  Program  Success 


Q# 

SENTENCE 

SUBCATEGORY 

CATEGORY 

4 

Communication  with  the  contractor  is  vital. 

You  have  to  have  a  good  relationship  with 
them.  You  won't  produce  anything  if  you  are 
lying  to  one  another.  If  the  contractor  feels 
comfortable  coming  to  the  government  and 
telling  them  the  truth,  we  just  have  the  major 
problem  we’re  going  to  have  schedule  slips  and 
then  you  can  work  through,  negotiate  your 
chance  is  I  think  of  successfully  working 
around  that  problem  much  greater  than  if  they 
hide  it  comes  out  later  on  at  the  program  you 
can’t  do  anything  about  it  so  that's  absolutely  a 
major  concern.  You  have  to  be  able  to  have 
open  communication. 

Communication 

Program  Success 
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One  way  to  detennine  the  importance  of  particular  categories  is  to  count  the 


number  of  occurrences  (Miles  &  Huberman,  1994).  Table  10  provides  this  count  data, 
and  gives  a  sense  of  the  relative  “significance”  of  particular  categories  and  subcategories. 


Table  10.  A  Count  of  the  Number  of  Subcategories  in  Each  Category 


SUBCATEGORIES 

CATEGORIES 

COUNT 

CLUSTERS 

Funding 

Challenges 

10 

CHALLENGES 

Schedule 

Challenges 

5 

Cost 

Challenges 

4 

Work  ethics 

Culture 

5 

Cultural  similarity 

Culture 

1 

Language 

Culture 

1 

Non-US  contractor 

Funding 

1 

Public  desire 

Funding 

1 

Dependent  to  contractor 

Traditional  Acquisitions 

1 

Expensive  logistics  network 

Traditional  Acquisitions 

1 

Limited  availability 

Traditional  Acquisitions 

1 

Statement  of  work 

Contract  Development 

3 

GOAL 

Defining  performance  metrics 

Contract  Development 

2 

Flexibility 

Contract  Development 

2 

Performance-based  acquisition 

Environment  of  the  Contract 

3 

Sub-contracting 

Environment  of  the  Contract 

3 

Different  procurement  type 

Environment  of  the  Contract 

2 

Communication 

Program  Success 

15 

Product  quality 

Program  Success 

14 

Customer  satisfaction 

Program  Success 

13 

Performance-based  acquisition 

Type  of  Contract 

2 

Alternative  service  delivery 

Type  of  Contract 

1 

Cost  plus  type  based  on 
performance 

Type  of  Contract 

1 

Award  fee 

PBA  metrics 

6 

SUCCESS 

BOOSTERS 

Cooperation 

PBA  metrics 

2 

Earned  value  management  system 

PBA  metrics 

2 
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Table  10  presents  the  counted  subcategories  in  each  category.  To  refine  a  new 
theoretical  perspective,  the  researcher  clustered  the  subcategories  according  to  their  role 
in  the  success  of  a  performance-based  acquisition  program.  This  delimiting  process  led 
to  three  new  theoretical  clusters:  challenges,  goal  definition,  and  success  boosters.  To 
provide  consistency,  each  subcategory  was  cross-checked  with  categories  accordingly. 

For  example,  there  were  10  sentences  in  the  subcategory  called  funding  within  the 
category  called  challenges.  Accordingly,  the  researcher  included  the  funding  subcategory 
in  the  challenges  cluster. 

Figure  7  presents  the  relative  significance  of  the  subcategories  based  on  this 
counting  process.  When  making  generalizations  or  saying  something  is  important, 
significant,  or  recurrent  we  have  to  come  to  an  estimate  by  making  counts  (Miles  & 
Huberman,  1994).  The  numbers  indicate  the  number  of  times  interviewees  repeated 
themes  in  each  subcategory.  For  example,  flexibility  appeared  six  times  in  the  Program 
success  category  and  twice  in  the  Contract  development  category  for  a  total  of  eight.  A 
detailed  list  of  the  subcategories  is  included  as  Appendix  C. 


Figure  7.  Numbers  of  Subcategories  of  the  Interview 
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Based  on  the  count  data,  it  is  apparent  that  defining  performance  metrics  was 
particularly  important  or  significant  to  the  interviewees.  Interviewees  acknowledged  that 
while  it  is  difficult  to  define  the  best,  most  fair,  and  most  reasonable  performance  metrics 
for  both  the  government  and  the  contractor,  defining  those  standards  is  the  key  to 
acquisition  success.  One  of  the  interviewee  indicates  this  as;  “The  difficulty  lies  in 
appropriately  defining  the  performance  standards  to  ensure  that  the  customer  gets  what  he 
needs  (within  cost  guidelines),  and  the  contractor  is  able  to  achieve  those  standards 
(within  the  resource  restrictions  applied)  while  still  making  a  fair  profit.” 


Challenges 


Goal 


Success 

Boosters 


Figure  8.  The  Schema  of  Clustered  Subcategories 
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By  clustering  subcategories  of  the  interviews  as  illustrated  in  Figure  8,  three 
factors  emerged  leading  to  the  success  of  the  perfonnance-based  acquisition.  Those  are 
clear  definition  of  the  goal,  challenges,  and  success  boosters.  The  goal  should  be  to 
acquire  the  desired  product  or  service  which  the  user  needs  met  within  the  reasonable 
amount  of  time  and  within  the  reasonable  amount  of  cost  that  will  be  beneficial  for  both 
parties.  Interviewees  see  that  the  most  important  metric  for  the  success  of  the  acquisition 
programs  as  customer  satisfaction.  One  of  the  interviewees  indicates  that  as  “When  you 
meet  the  requirements  of  the  contract,  when  you  meet  the  original  intent  of  your  design  of 
your  product  and  it's  been  successfully  used  without  major  failures,  I  think  that  is  a 
success.” 

The  second  factor  that  emerged  is  the  ability  to  recognize  and  respond  to 
challenges  in  PBA.  Challenges  may  be  related  to  the  nature  of  the  contract  such  as 
whether  it  is  a  software  development  program  or  a  research  program,  etc.  Socio-cultural 
issues  such  as  different  work  ethics,  languages,  or  politics  also  emerged  as  challenges  for 
acquisition  programs.  The  most  important  challenges  that  emerged  from  the  clustered 
subcategories  were  funding,  security,  vanishing  vendors,  work  ethics  in  different 
countries,  language,  and  unstable  user  needs.  The  interviewees’  most  repeated  challenge 
to  program  success  was  funding.  One  interviewee  talks  about  funding  as  “The  hard  part 
at  least  with  the  American  acquisitions  is  that  in  my  opinion  we  have  under  budgeted 
every  program.” 

Finally,  the  third  factor  that  emerged  is  success  boosters.  In  complex  acquisition 
programs,  there  is  always  potential  for  ambiguities.  For  example,  the  business 
relationship  between  the  government  organizations  and  the  contractor  may  not  be  clear. 
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Or,  the  documentation  prepared  by  the  government  organization  may  not  be  understood 
by  the  contractor.  In  order  to  reduce  ambiguity  and  increase  performance  in  a 
performance-based  acquisition  environment,  this  study  demonstrates  some  factors  that 
assist  the  success  of  the  program.  These  success  boosters  emerged  as  defining 
performance  metrics,  communication,  establishing  integrated  PBA  teams,  working  with 
experienced  contractor,  training,  flexibility,  and  periodical  reviews  of  the  program.  One 
critical  success  booster  for  the  acquisition  program  is  communication.  An  interviewee 
talks  about  the  importance  of  communication  as  “But  again,  it  really  comes  down  to  your 
team,  the  customer,  government,  and  the  contractor.  As  long  as  everybody  works 
together  and  has  good  communication  and  trust,  I  think  you  are  going  to  have  success 
whether  you  are  using  PBA  or  not.” 

The  subcategory  data  were  also  “rolled  up”  into  categories.  These  categories 
helped  to  draw  conclusions  from  the  patterns  of  the  coded  data.  The  purpose  of  the 
grounded  theory  method  is  to  build  a  theory  or  write  a  story  from  the  categories  or 
subcategories  that  are  applicable  to  and  indicated  by  the  data  under  the  study  (Glaser  & 
Strauss,  1967).  Using  the  constant  comparative  method  (Locke,  2003),  the  researcher 
developed  categories  in  order  to  create  a  theory  from  the  interview  data.  The  categories 
that  emerged  were  program  success,  challenges,  PBA  metrics,  contract  environment, 
contract  development  process,  contract  type,  culture,  and  traditional  acquisitions.  Those 
categories  were  created  according  to  the  primary  meanings  of  sentences.  Interviewees 
spoke  mostly  about  program  success.  Specifically,  interviewees  discussed  the  factors 
that  can  facilitate  the  performance  of  the  program,  the  obstacles  to  success  of  the 
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program,  and  the  metrics  for  determining  the  success  of  the  program.  The  percentage  of 
interview  categories  is  illustrated  in  Figure  9. 


■  Traditional  Acquistions 

■  Culture 

■  Type  of  Contract 

■  Contract  development 

■  Environment  of  the  Contract 

■  PB  A  metrics 

■  Challenges 

■  Program  Success 


Figure  9.  Categories  of  the  Interviews 


The  most  important  metric  for  program  success  is  the  product  quality  or  customer 
satisfaction.  Only  one  out  of  thirteen  interviewees  chose  the  number  of  deliveries  as  the 
most  important  metric.  Twelve  of  the  interviewees  said  that  the  most  important  metric 
for  determining  the  success  is  satisfying  user  needs.  Although  customer  satisfaction  is 
the  most  important  performance  metric  for  success,  other  factors  such  as  product  delivery 
date  and  the  number  of  deliveries  clearly  affect  customer  satisfaction,  at  least  indirectly. 
If  the  users  are  not  satisfied  within  the  operational  need  time  with  the  correct  amount  of 
material  then  we  cannot  speak  about  the  success  of  the  acquisition. 
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It  is  difficult  to  draw  conclusions  about  the  effects  of  cultural  differences  in 


performance-based  acquisition,  as  the  researcher  could  only  conduct  three  interviews 
with  individuals  outside  the  US.  Nevertheless,  the  findings  related  to  cultural  differences 
seem  to  be  mostly  about  security  issues,  rather  than  the  ability  to  execute  the  method 
properly.  Clearly,  nations  want  to  protect  their  technological  advantage  from  other 
countries  whether  those  countries  are  western  or  not.  Another  cultural  issue  that 
appeared  to  be  significant  was  potential  for  different  work  ethics  to  interfere  with 
performance.  In  a  performance-based  acquisition  environment,  with  appropriate 
regulations,  well-defined  performance  metrics,  and  with  communication  and 
collaboration  this  issue  might  be  mitigated. 

Nearly  all  of  the  interviewees  (85  percent,  including  all  three  non-U. S. 
interviewees)  agreed  that  performance-based  acquisition  might  be  successful  in  a  non¬ 
western  environment.  Some  interviewees  expressed  their  belief  that  implementation  of 
PBA  would  be  easier  in  a  more  similar  culture  because  work  ethics  and  language  tend  to 
be  similar.  On  the  other  hand,  as  another  interviewee  points  out,  language  barriers  might 
occur  even  in  the  same  cultural  environment.  Language  barriers  and  different  work 
ethics  might  be  mitigated  through  frequent  communication,  constant  cooperation,  and 
clear  regulation.  The  percentage  of  the  interviewees  who  think  PBA  can  be  successful  in 
a  non- Western  environment  is  shown  in  Table  11. 

Table  11.  Expected  Success  Percentage  of  PBA  in  non-Westem  environment 


Can  PBA  be  Successful  in  a  Non-Westem  Country? 

YES 

11  (85%) 

NO 

- 

NOT  SURE 

2  (15%) 
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Summary 


The  aim  of  this  chapter  was  to  perform  the  analysis  and  present  results  of  the 
research.  The  chapter  covered  the  analysis  of  the  qualitative  data  gathered  from  the  semi- 
structured  interview  questions.  Although  the  study  was  limited  to  thirteen  interviewees  and 
only  three  of  them  were  from  different  nationalities,  the  results  offered  insights  into  the 
success,  benefits,  and  challenges  of  implementing  performance-based  acquisition  in 
different  cultural  environments.  The  sentences  were  coded  and  counted  to  see  the 
significant  patterns  leading  to  answers  to  the  research  questions.  Three  factors  of  PBA 
success  emerged  from  the  categories  of  the  interview  data.  These  three  factors  were  clear 
goal  definition,  challenges,  and  success  boosters.  Taking  into  consideration  these  factors, 
successful  implementation  of  PBA  in  non- western  countries  seems  to  be  plausible. 
Managing  cultural  diversity  and  defining  the  performance  standards  upfront  would  be 
important  considerations  prior  to  attempting  performance-based  acquisition  outside  the 
United  States.  The  final  chapter  will  draw  the  conclusions,  present  limitations  of  the 
study,  and  make  recommendations  for  future  research. 
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V.  Conclusions 


Introduction 

The  primary  objective  for  this  research  was  to  explore  the  opportunities  and 
challenges  of  implementing  performance-based  acquisition  in  different  acquisition 
environments.  The  developments  in  infonnation  technology,  transportation,  and  supply 
chain  environments  make  the  world  more  flat  than  ever.  Business  and  government 
relationships  are  becoming  more  integrated.  “It  seems  reasonable  to  expect  that  many  of 
the  existing  service  industries  will  continue  as  productive  as  any  in  the  world,  while  new 
industries  will  quickly  leam  to  operate  to  world-class  standards  of  efficiency”  (McRae, 
1994,  p.212).  It  seems  likely  that  more  communication  and  collaboration  driven  business 
relationships  will  lead  to  new  acquisition  approaches.  Perfonnance-based  solutions  are 
one  of  the  more  popular  newer  approaches.  Although  most  of  the  performance-based 
acquisition  programs  are  in  western  countries,  such  as  the  United  States,  non-Western 
countries  are  likely  to  follow  this  trend  as  well.  Using  qualitative  research  techniques, 
this  study  investigated  the  factors  that  would  make  PBA  successful  in  a  different  culture. 
The  research  seeks  to  answer  the  question:  “can  PBA  be  successful  in  a  non-western 
acquisition  environment?” 

This  chapter  will  present  the  conclusions  of  this  study,  identify  the  limitations  to 
the  study,  and  offer  recommendations  for  future  study. 
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Conclusions 


Performance-based  acquisition  creates  a  new  acquisition  environment  for  both 
industry  and  government  by  promoting  flexibility  and  innovation  and  by  creating  win- 
win  solutions  through  effective  communication.  It  is  no  longer  optional;  it  is  a  business 
necessity  in  both  public  and  the  business  sectors  (Garrett,  2005).  Performance-based 
acquisition  is  mostly  implemented  in  western  countries.  The  results  of  selected  PBA 
programs  show  that  this  performance-based  approach  is  becoming  more  popular  because 
the  perception  of  acquisition  professionals  is  that  it  creates  better  performing  systems, 
more  cost  efficient  programs,  and  rapid  solutions  to  problems  (DAU,  2005).  The  purpose 
of  this  study  was  to  investigate  whether  perfonnance-based  acquisition  would  be 
successful  or  not  if  implemented  in  a  non-western  environment.  According  to  the  results 
presented  in  Chapter  IV,  the  answer  is  a  qualified  “yes.”  Most  of  the  thirteen 
interviewees  did  not  have  direct  experience  implementing  PBA  in  a  non- Western 
environment.  However,  while  acknowledging  that  there  may  be  cultural  and 
socioeconomic  challenges  to  overcome,  most  of  them  agreed  (11  of  13)  that  PBA  would 
be  successful  if  implemented  in  a  non-westem  environment. 

Certain  cultural  differences  may  affect  the  success  of  a  perfonnance-based 
acquisition.  For  instance,  interviewees’  comments  centered  on  cultural  issues  such  as 
language  barriers,  work  ethic  differences,  etc.  For  example,  regarding  language  barriers 
one  interviewee  stated,  “I  mean  there  is  a  language  banier  but  within  the  contract  you'll 
define  what  language  you're  going  to  convey  them.  All  the  contracts  will  be  in  English 
not  Turkish  because  we  can't  read  Turkish.  And  all  the  technical  publications  will  be 
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English.”  Another  challenge  to  implementation  of  PBA  in  a  non-Westem  context  is  the 
effect  that  different  work  ethics  might  have  on  a  project.  The  local  knowledge  of  cultural 
differences  can  be  managed  advantageously  in  PBA  acquisitions  because  the  “how”  is 
left  up  to  the  contractor.  An  interviewee  indicates  that  as  “Having  a  non-US  contractor 
they  have  experienced  of  what  is  needed  in  that  location  and  what  type  of  methods  they 
need  to  use  for  construction  and  things  like  that.  The  advantage  is  definitely  to  the  non- 
US  contractor  in  that  situation.”  On  the  other  hand,  one  of  the  interviewees  mentioned 
how  cultural  similarity  makes  it  easier  to  implement  PBA  in  a  different  environment. 
“The  closer  the  country  is  to  your  culture,  the  easier  it  is.  You  know  when  you  are  in, 
let’s  say  in  an  Anglo-Saxon  environment  where  the  work  ethic  tends  to  be  the  same,  the 
language  is  similar;  the  way  you  communicate  is  similar.  I  think  it  is  probably  easier.” 

One  of  the  primary  objectives  of  the  study  was  to  investigate  whether  to 
implement  performance-based  acquisition  in  non- western  countries.  Understanding  the 
similarities  and  differences  between  countries  would  help  acquisition  professionals  to 
implement  PBA  in  non-western  countries.  This  study  confirmed  that  cultural  factors 
could  play  at  least  a  minor  role  in  the  success  of  performance-based  acquisitions. 
Communication,  cooperation,  and  more  integrated  business  relationships  could  mitigate 
the  risk  of  cultural  challenges  affecting  the  success  of  PBA.  This  research  also  reveals 
that  both  industry  and  government  can  successfully  implement  performance-based 
acquisition  approach  for  the  incoming  acquisition  programs  regardless  of  the  cultural 
differences. 

Another  objective  of  the  study  was  to  uncover  the  critical  factors  affecting  the 
success  of  PBA  programs.  Understanding  these  critical  factors  would  help  acquisition 
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professionals  implement  PBA  successfully  in  general,  whether  in  the  U.S.  or  outside  the 
U.S.  This  study  produced  three  critical  factors  for  the  success  of  performance-based 
acquisition.  Both  government  and  industry  would  benefit  of  implementing  PBA  by 
defining  the  goal  clearly,  managing  challenges  effectively,  and  operating  the  success 
boosters  accurately. 

Investigating  the  research  question  and  analyzing  the  gathered  interview  data 
explains  the  success  of  the  perfonnance-based  acquisition  in  non-westem  countries.  In 
this  century,  the  world  is  getting  flat  and  accessing  the  infonnation  is  easier  than  before. 
Implementing  a  successful  approach  in  a  different  environment  only  requires  applying 
the  same  rules  or  similar  rules  to  adapt  to  the  cultural  diversity.  Finally,  this  study  gives 
some  insight  about  the  success  of  the  PBA  in  a  non-westem  country.  Organizations  in 
the  public  or  private  sector  should  pick  an  acquisition  approach  that  would  promote 
flexibility  and  innovation  and  create  win-win  solutions  for  both  buyer  (s)  and  seller  (s) 
through  effective  communication  and  organizational  goal  alignment.  In  order  to  achieve 
internal  or  external  organizational  targets,  using  PBA  seems  to  be  the  future  approach  in 
acquisition  environment. 

Limitations 

Limitations  restrict  the  findings  of  this  study.  For  instance,  many  programs  that 
use  performance-based  methods  belong  to  the  defense  industry,  and  often  infonnation 
from  these  programs  is  either  classified  or  highly  proprietary  in  nature.  It  was  difficult  to 
obtain  quantitative  information  such  as  schedule  and  cost  perfonnance.  However, 
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interview  data  partially  mitigated  this  limitation  by  providing  qualitative  information  that 
could  act  as  a  proxy  for  quantitative  infonnation.  Another  limitation  was  the  inability  to 
make  direct  comparisons  of  alternative  acquisition  methods.  Organizations  choose  only 
one  acquisition  method  for  a  program  so  the  information  needed  to  compare  the  success 
of  the  PBA  with  alternative  methods  in  the  same  context  was  not  available. 

Additionally,  the  study  was  limited  to  10  U.S.  (and  3  non-U. S.)  acquisition 
officers  serving  at  Wright  Patterson  Air  Force  Base,  as  time  constraints  precluded  the 
possibility  of  gathering  data  from  as  broad  a  sample  of  acquisition  professionals  as  would 
have  been  preferred.  Only  three  out  of  the  thirteen  interviewees  were  not  from  the  United 
States.  Thus,  the  research  questions  related  to  the  international  context  are  difficult  to 
address  with  rigor.  Moreover,  none  of  the  interviewees  was  from  a  non-military 
environment,  and  none  was  a  contractor. 

Like  all  qualitative  studies,  the  methods  used  to  identify  related  responses  and  to 
detennine  the  themes  within  the  responses  were  subjective  and  relied  on  the  researcher’s 
knowledge  and  opinion.  These  methods  may  inject  researcher  bias  into  the  results  of  the 
study.  One  procedure  to  reduce  researcher  bias  in  these  cases  is  to  have  more  than  one 
researcher  independently  code  the  interviews,  and  compare  the  results  for  inter-rater 
reliability.  There  was  not  time  to  accomplish  this  procedure  for  this  study.  Finally,  the 
approach  used  for  this  study  may  limit  the  ability  of  future  researchers  to  repeat  the 
results  of  this  study. 
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Future  Research 


Future  research  that  conducts  interviews  from  a  wide  variety  of  different 
nationalities  is  recommended.  Conducting  interviews  with  acquisition  professionals  from 
a  wide  variety  of  different  countries  and  different  sectors  (industry  and  government) 
would  enhance  the  rigor  of  the  findings.  In  addition,  increasing  the  number  of 
interviewees  from  different  nationalities  may  improve  the  ability  to  answer  research 
questions  related  to  the  international  context.  Talking  with  contractors  to  get 
perspectives  from  both  sides  of  the  contract  would  increase  understanding  of  the  research 
question. 

This  study  chose  a  qualitative  approach  to  investigate  the  research  question  due  to 
difficulty  attaining  quantitative  data.  Future  research  might  use  both  qualitative  and 
quantitative  methods  to  improve  the  research  design.  The  use  of  mixed  methods  might 
enhance  the  findings  related  to  cultural  differences  in  particular.  Future  studies  may 
replicate  the  findings  of  this  study,  increasing  the  validity  of  this  research. 

Summary 

The  purpose  of  this  research  was  to  investigate  how  the  cultural  differences  might 
affect  the  success  of  performance-based  acquisition  in  non-western  countries  and  to 
detennine  common  factors  that  lead  to  success  and  failure.  The  coding  of  categories  and 
subcategories  from  the  interviews  illustrated  common  themes.  These  themes  allowed  the 
researcher  to  develop  new  theory  related  to  performance-based  acquisition.  The  primary 
objective  question  was  answered  by  the  story  that  emerged  from  the  patterns  of  the  coded 
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sentences.  Ultimately,  the  research  supports  the  view  that  the  performance-based 
acquisition  approach  is  applicable  to  acquisitions  in  different  cultural  environments. 
Common  concerns  such  as  inadequate  security,  difficult  language  barriers,  or 
incompatible  work  ethics  or  culture  may  be  mitigated  through:  1)  appropriate  training,  2) 
setting  international  regulations  and  standards,  3)  better  communication,  4)  properly 
sharing  risk,  and  5)  effective  collaboration. 
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Appendix  A.  Semi-structured  Interview  Questions 


Demographic  Information:  Name,  rank,  nationality,  current  job  title,  current  job 

description. 

Relevant  Experience: 

How  much  (and  what  kind  of)  work  experience  do  you  have  in  acquisition? 

Do  you  have  a  basic  understanding  of  perfonnance  based  acquisition  methods? 

Semi-structured  Interview  Questions: 

1 .  Have  you  ever  been  involved  in  a  contract  using  perfonnance  based  acquisition 
methods?  If  so,  can  you  describe  the  methods  that  were  used?  What  “worked”? 
What  did  not  “work”?  In  your  estimation,  did  performance  based  methods  result 
in  cost,  schedule,  or  performance  benefits  compared  to  other  methods? 

2.  Have  you  been  involved  with  non-U. S.  contracts?  If  so,  did  they  use  performance 
based  acquisition  methods?  If  so,  could  you  describe  how  those  methods  were 
implemented?  Would  you  say  that  the  methods  were  implemented  substantially 
the  same  as  they  would  be  for  a  U.S.  contract?  Substantially  different? 

3.  In  the  future,  do  you  plan  to  implement  performance  based  acquisition  methods 
(in  a  U.S.  environment  or  in  a  non-U. S.  environment)?  Do  you  anticipate  any 
advantages?  Disadvantages? 

4.  In  acquisition  programs  in  which  you  have  been  involved,  what  has  been  the  most 
critical  issue  or  obstacle  to  success  (e.g.  cost,  sustainability,  meeting  the  user’s 
requirements,  schedule,  etc.)? 

5.  Can  you  share  an  acquisition  “horror  story”  with  me,  when  something  went 
wrong? 

6.  Can  you  share  an  acquisition  “hero  story”  with  me,  when  something  went  right? 

7.  When  can  you  describe  an  acquisition  as  successful?  What  are  the  most  important 
metrics  for  determining  success? 
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Appendix  B.  Coding  of  Responses 


Q# 

SENTENCE 

SUBCATEGORY 

CATEGORY 

1 

I  would  say  last  contract  I  did  used  PBA 
methods.  I  don’t  think  it  was  explicitly  sets  but 
we  did.  The  performance  was  the  most 
important  part  what  we  were  doing  because  it 
was  a  research  program. 

Research  program 

Type  of 
Program 

1 

We  used,  first  of  all,  earned  value  management 
system.  I  think  that’s  one  of  the  methods  with 
the  talk  about  where  you  figure  out  how  you 
progress  is  based  on  the  amount  of  work  you 
are  supposed  to  accomplish  versus  how  much 
you  actually  accomplish  and  so  on. 

Earned  value 
management 
system 

PBA  metrics 

1 

We  had  weekly  program  management  reviews 
where  we  would  be  told  with  the  status  was, 
any  sort  of  the  problems  that  they  run  in  to 
see  how  well  they've  been  performing  on  the 
contract. 

Periodical  reviews 

PBA  metrics 

1 

If  there  were  issues  than  we  would  go  ahead 
and  try  to  figure  out  how  much  cost  to 
remedies  those  issues  and  gain  the  performance 
that  we  want  out  of  the  contract. 

Periodical  reviews 

PBA  metrics 

1 

Generally  we,  having  the  weekly  meetings 
were  good,  we  had  very  open  relationship  with 
the  consortium  that  was  developing  the 
program. 

Communication 

PBA  metrics 

1 

So,  really  the  best  thing  we  were  having;  you 
have  articles  that  are  delivered,  those  were  the 
things  that  you  can  assess  how  well  they're 
performing. 

Documentation 

Program 

Success 

1 

So,  generally  because  we  had  such  as  open 
relationship,  it  was  very  like  flowing  kind  of 
thing. 

Open  relationship 

Program 

Success 

1 

The  earned  value  system  worked  pretty  well. 

Earned  value 
management 
system 

PBA  metrics 

1 

They  were  very  good  giving  us  reports  we 
knew  when  there  was  going  to  be  a  cost  issue 
and  we  were  able  to  work. 

Periodical  reports 

Program 

Success 

1 

I  think  generally  the  best  thing  about,  it  was  the 
team  was  very  cohesive,  and  we  communicated 
well  and  it  was  not  an  adversarial  kind  of 
relationship. 

Establishing  teams 

PBA  metrics 
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Q# 

SENTENCE 

SUBCATEGORY 

CATEGORY 

1 

What  did  not  work?  It  is  difficult.  I  mean  it 
depended  on.  Scheduling  was  always  difficult, 
because  it  takes  the  contractors  a  long  time  to 
generate  a  schedule. 

Schedule 

Challenges 

1 

So,  to  use  the  program  schedule  as  a  tool  could 
be  difficult. 

Schedule 

Challenges 

1 

I  mean  we  had  to  go  through.  You  know  if  it 
took  a  major  rewrite  if  there  was  some  major 
problems  so  that  could  have  gone  better  as 
certainly  when  you  would  have  a  schedule  slip 
of  a  major  cost  group  that  would  require  you  to 
redo  certain  things  or  look  at  the  task  and 
figure  out  what  needed  to  be  ultimate. 

Periodical  reviews 

Program 

Success 

1 

Everyone  had  a  good  idea  what  needed  to  be 
done.  But  making  the  schedule  reflect  that 
could  be  difficult,  sometimes  that  would  take 
some  major  work  after  various  decision  points. 

Schedule 

Challenges 

1 

Okay  now  they  have  to  go  crank  the  machine 
again,  get  us  a  new  schedule.  Those  things 
could  have  been  better  I  would  think. 

Schedule 

Challenges 

1 

Otherwise  as  far  as  the  way  that  we  were,  I 
mean  the  performance  was  very  good  we  were 
getting  what  we  wanted  out  of  the  contract 
certainly  all  the  way  through  the  time  I  left. 

Product  Quality 

Program 

Success 

1 

We  had  a  cost  plus  kind  of  a  contract  where 
you  pay  based  on  the  performance  and  so, 
certainly  that  allowed  us  to,  you  know  this  was 
a  research  program  there  were  a  little  different 
from  acquisitions.  It  is  condensed  and  we  were 
looking  for  some  specific  things 

Cost  plus  type 
based  on 
performance 

Type  of 
Contract 

1 

So,  because  the  way  contract  was  written  it 
allowed  us  a  lot  of  flexibility.  If  we  decided  for 
whatever  reason  that  we  did  not  if  we  could 
remove  items  from  the  contract  very  easily,  or 
we  could  say  we  don't  have  enough  money  to 
do  this. 

Contract  flexibility 

Program 

Success 

1 

So,  we  would  get  either  bring  some  of  the  work 
in  house,  in  our  own  labs  so,  we  could  save 
money.  So,  we  were  able  to  manipulate  the 
contract  much  cheap  much  more  easily. 

Flexible  work 
environment 

Program 

Success 

62 


Q# 

SENTENCE 

SUBCATEGORY 

CATEGORY 

1 

We  were  on  top  of  them  all  the  time  to  make 
sure  that  they  were  using  what  is  called 
management  reserve  to  put  out.  Management 
reserve  is  that  amount  of  contract  that  is  left 
after  the  cost  that  we  ordinarily  go  to  incentive. 

Authority 

Program 

Success 

1 

So,  you  specify  this  is  your  reserve  and  you 
have  that  to  put  out  fires  as  we  say  to  solve 
problems.  And  the  fewer  problems  you  have 
the  more  you  keep  at  the  end.  So,  the  program 
was  very  good  and  identifying  that 
management  reserve  and  having  the 
contractors  to  use  that  to  prevent  cost  groups  so 
in  that  case  absolutely. 

Management 

Reserve 

Program 

Success 

1 

We  were  able  to  keep  cost  down  a  lot  more  and 
because  we  were  very  knowledgeable  what 
needed  to  be  done  if  we  saw  opportunities  to 
reduce  cost  by  using  different  methods  than  we 
could  easily  remove  scat  lines  from  the 
contract  that  didn’t  have  another  organization 
do  those. 

Flexible  contracting 

Program 

Success 

1 

Yeah  It  does  save  some  money. 

Cost 

Program 

Success 

1 

I  mean  we  still  have  cost  growth  because  you 
save  money  here  and  there  but  because  the  way 
we  estimate  the  work  you  know  you  use 
historical  data  and  on  the  research  program  that 
is  difficult  because  it's  brand  new  project  so 
you  don’t  always  know  exactly  what  your 
unexpected  cost  will  be.  Instead  of  let’s  say 
traditional  acquisition  programs  where  they 
have  still  cost  growth  a  lot  of  times  but  it's 
maybe  sometimes  it's  not  as  difficult  to  project 
what  you  are  doing. 

Enough  historical 
data 

Challenges 

1 

So,  we  did  saved  money,  the  program 
expensed  the  growth  but  we  also  added  fly,  we 
added  performance  because  of  that.  So  it  was 
very  reasonable  of  the  way  that  it  worked  out. 

Cost 

Program 

Success 

1 

We  definitely  had  very  good  perfonnance  in 
the  wind  tunnels,  everything  with  the  engines, 
and  all  that  kind  of  stuff. 

Product  Quality 

Program 

Success 

1 

As  far  as  schedule  certainly  we  were  able  to 
maintain  schedule  very  well. 

Schedule 

Program 

Success 
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Q# 

SENTENCE 

SUBCATEGORY 

CATEGORY 

1 

But  they  had  caused  some  slips  just  because 
like  facilities  don't  always  work  out.  Things 
like  that.  So,  unexpected  things  caused 
schedule,  but  overall  the  schedule  was  very 
well  at  here  too. 

Facilities 

Problems 

1 

It  was  a  unique  approach  the  contract  was  a  lot 
different.  It's  called  other  transactions 
agreement  (OTA). 

Other  Transactions 
agreement 

Type  of 
Contract 

1 

So,  we  had  a  more  integrated  contractor 
instead  of  having  a  traditional  prime-sub 
relationship. 

Integration 

Program 

Success 

1 

The  contractors  were  put  together  in  a 
consortium  which  means  that  both  of  them 
were  equal  foot-in.  One  would  be  the 
administrative  manager  but  otherwise  the 
government  and  this  case  two  main  contractors 
were  equal  foot  in  to  negotiate. 

Consortium 

Program 

Success 

1 

We  saved  a  lot  because  nobody  could  blame 
the  other  for  slowing  the  program  down  and  so 
on.  It  was  so  much  more  collaborative 
environment. 

Collaboration 

Program 

Success 

1 

I  think  in  that  case  we  certainly  did  save  a  lot 
of  schedule. 

Schedule 

Program 

Success 

1 

Performance  was  very  high  level.  Because  they 
were  working  together  in  this  case  and  the  way 
that  they  execute  the  program  was  very  good. 

Collaboration 

Program 

Success 

1 

But  until  the  vehicle  flies  we  won’t  know  how 
well  we  achieved  because  it's  brand  new  item. 
So,  again  it's  difficult  there  is  no  historical 
comparison  for  this  kind  of  project. 

Enough  historical 
data 

Challenges 

2 

We  didn’t  have  non-US  contractor,  although 
suppliers  for  various  parts  might  be  non-US  so 
they  would  be  subcontract.  We  did  have  one  or 
two  of  those. 

Sub-contracting 

Environment  of 
the  Contract 

2 

Where  we  were  working  with  the  company 
there  was  originally  US  and  then  they  were 
bought  foreign  firm.  For  example  like  Allison 
engines  were  bought  by  Rolls  Royce  engines 
sometime  ago. 

Sub-contracting 

Environment  of 
the  Contract 

2 

There  are  issues  as  far  as  security  which  have 
you  know  especially  when  you  are  working 
with  munitions  or  technical  systems. 

Security 

Challenges 
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2 

So  my  experience  would  be  more  in  that  line  as 
far  as  you  know  the  way  that  you  do  work  in 
performance.  I  am  not  sure  there  would 
probably  be  some  effects.  I  mean  the  work 
ethics  in  different  countries,  different  work 
hours,  different  laws,  labor  and  other  things.  So 
that  cause  certain  things  but  otherwise  I  don't 
know  too  much  about  that. 

Work  ethics 

Environment  of 
the  Contract 

2 

Security  is  big  one.  Especially  when  you  are 
dealing  with  engine  technology  because  it  is  a 
technology  that  the  nations  like  to  protect. 

Security 

Challenges 

2 

So,  when  you  deal  with  a  company  that  is  part 
of  a  country  that  is  looking  to  do  some  similar 
work  in  the  research  that  you  are  doing  it's 
more  difficult  to  exchange  information  because 
sometimes  like  a  lot  of  times  for  example  when 
Rolls  Royce  bought  Allison  they  had  to  design 
their  building  such  a  way  that  all  the 
employees  of  Allison  who  were  all  US  citizens 
could  work  on  their  programs  and  not  have 
issues  with  security  because  they  were  all 
cleared  US  personnel. 

Exchanging 

Information 

Challenges 

2 

So,  it  was  a  subsidiary  of  Rolls  Royce.  And  the 
Rolls  Royce  management  could  not  gain  some 
of  that  information  because  they  were  British 
firm.  So,  some  of  that  was  they  were  not  party 
to. 

Sub-contracting 

Environment  of 
the  Contract 

2 

And  the  same  issue  of  other  companies  that 
way.  So,  for  example  if  you  a  company  that's  a 
part  of  a  known  competitor  a  lot  of  times  you 
can’t  exchange  certain  information  the 
government  might  consider  secret  so  on  so. 

Exchanging 

Information 

Challenges 

2 

That  adds  complication  so  that  adds  cost 
because  the  more  security  is  very  expensive. 

Security 

Cost 

2 

The  way  that  we  protect  things  requires  a  lot  of 
energy  and  it  is  expensive.  So  that’s  one  issue 
very  common. 

Security 

Cost 
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Yeah  I  think  that  makes  sense.  GSA  (General 
Services  Administration)  certainly  likes  to  use 
Performance  Based.  They  recommend  that.  It 
is  sort  of  slow  progression  one  when  they  talk 
about  Performance  based  contract  and  PBA. 
Those  things  are  definitely  useful  and  again 
since  really  what  you  want  out  of  a  contract 
just  not  the  spend  money  you  want  quality 
product  so  performance  is  probably  everything. 
So,  I  would  say  yes. 

Product  Quality 

Program 

Success 

3 

I  guess  it  would  depend  on  the  contract.  I  mean 
if  the  country  that  you  are  working  with  uses 
the  same  methods  of  labor  and  everything  else 
you  can  do  that.  If  you  work  with  a  country 
that  has  let’s  say  restrictions  on  the  numbers  of 
hours  they  can  work  in  a  day  something  like 
that.  It  might  be  a  little  more  difficult.  It  would 
really  have  to  depend  on  the  relationship  with 
the  country  and  how  the  federal  government 
has  agreements  set  up. 

Different 
procurement  type 

Environment  of 
the  Contract 

3 

I  think  the  closer  the  countries  to  your  culture 
are  easier  it  is  to.  You  know  when  you  are  in, 
let’s  say  in  an  Anglo-Saxon  environment 
where  the  work  ethic  tends  to  be  same,  the 
language  is  similar;  the  way  you  communicate 
is  similar.  I  think  it  is  probably  easier. 

Closer  better 

Culture 

3 

If  you  are  working  with  the  Japanese  company, 
a  company  that  is  very  focused  on  performance 
very  focused  on  customer  satisfaction  then  that 
would  be  easier. 

Performance 

Oriented 

Culture 

3 

You  know,  other  countries  might  have  a  little 
bit  more  relaxed  work  ethics  if  that's  the  case 
then  you  would  need  to  use  some  kind  of 
different  method.  I  think  performance  is 
always  a  good  way  to  go  but  sometimes  it 
might  be  difficult  to  measure  based  on  cultural 
differences. 

Work  ethics 

Culture 

3 

Again  without  knowing  a  lot  about  it,  it'd  be 
difficult  to  say  more  but  that's  what  I  would 
think.  As  far  as  working  with  a  foreign  country 
having  shorter  work  days  and  maybe  different 
attitude  might  be  a  little  bit  different. 

Work  ethics 

Culture 
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I  think  it  is  for  anybody  that  you'd  work  with  is 
advantage  by  means  of  PBA.  As  long  as  you 
can  trust  the  performance  is  being  reported  is 
accurate  then  yes. 

Performance 

Oriented 

Environment  of 
the  Contract 

4 

I  would  say  generally  it  has  more  to  do  with  I 
mean  if  you  are  working  with  technological 
systems  certainly  things  that  come  up 
technology  issues  that  you  know  you  are 
working  with  a  new  things  that  are  not  well 
tested  so  those  things  would  certainly  I  think 
be  major  issues. 

Technological 

systems 

Challenges 

4 

Otherwise  I’d  say  funding.  Funding  is  the 
major  concern.  Because  the  way  that  we 
receive  funding  in  the  govermnent  is  every 
year  congress  votes.  President  submits  the 
budget  congress  votes  on  it  and  then  they 
manipulate  the  budget  so  you  may  not  always 
get  the  budget  that  you  would  prefer  to  have  to 
do  your  program. 

Funding 

Challenges 

4 

That  is  the  major  thing  that  getting  the  correct 
amount  of  funding  and  getting  it  when  you 
need  it.  Because  sometimes  they'll  give  you 
money  but  they'll  ask  you  to  spend  it  but 
certain  times  programs  which  don’t  work  for 
the  performance  of  the  program.  This  is  I  think 
particularly  important  you  know  usually  in  a 
program  the  majority  of  the  work  is  done  when 
you're  doing  when  you're  building  and  you 
know  you  do  major  part  of  the  research  very 
front  part  of  the  program  And  then  sometime 
they'll  give  you  money  towards  the  end  of  the 
program.  It  is  not  as  useful.  That's  the  major 
obstacle.  When  you  get  your  money  and  then 
how  well  you  are  executing  and  how  well  you 
are  performing  because  the  way  they  ask 
performance  is  on  your  execution  how  much 
you  spent.  It  is  not  how  much  how  much  work 
has  been  done  necessarily 

Funding 

Discontinuity 

Challenges 
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4 

I  think  that's  the  major  thing.  That's 
discontinuity  between  how  you're  funded  and 
how  you're  expected  to  work  to  actually 
accomplish  something.  That  Sometimes  they'll 
cut  the  money  because,  oh  you  didn’t  spend 
enough,  you  didn't  produce  enough,  you  didn’t 
spend  enough.  So  in  that  case  performance 
methods  would  be  much  better  if  congress  look 
at  those,  but  they  don't  tend  to  look  at  those,  it's 
more  financial  they  look  at. 

Funding 

Discontinuity 

Challenges 

4 

Communication  with  the  contractor  is  vital. 

You  have  to  have  a  good  relationship  with 
them.  You  won’t  produce  anything  if  you  are 
laying  one  another.  If  the  contractor  feels 
comfortable  coming  to  the  government  and 
telling  them  the  truth  we  just  have  the  major 
problem  we’re  going  to  have  schedule  slips  and 
then  you  can  work  through,  negotiate  your 
chance  is  I  think  of  successfully  working 
around  that  problem  much  greater  than  if  they 
hide  it  comes  out  later  on  at  the  program  you 
can’t  do  anything  about  it  so  that's  absolutely 
major  concern.  You  have  to  able  to  have  open 
communication. 

Communication 

Program 

Success 

4 

Working  the  schedule.  You  know  making  sure 
you  have  accurate  schedule  keeping  you  track 
of  that,  keeping  track  of  hours,  and  keeping 
track  of  your  own  values  is  very  important  as 
well. 

Schedule 

Program 

Success 

4 

User  requirements.  The  best  is  when  they're 
stable.  If  the  user  requirements  change  then 
you'll  have  much  harder  time  accomplishing 
your  objectives.  And  they  do  tend  to  change.  . 

Customer 

satisfaction 

Program 

Success 

4 

And  the  longer  the  program  goes  the  more 
likely  it  is.  They'll  want  to  change  something 
because  they'll  develop  some  new  need  and 
they  want  to  insert  it  in  the  program  I  think  the 
quicker  you  do  the  program  the  better  your 
result  will  be. 

Longevity 

Program 

Success 
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4 

The  more  stable  your  user  needs  are  the  better. 
And  the  if  you  satisfy  those  needs  and  then 
your  user  will  be  happy.  Then  they  will  help 
your  funding  and  everything  else.  So  it's  very 
important  to  satisfy  customer.  Otherwise 
there's  no  reason  to  do  a  program  if  you  are  not 
satisfying  the  person  you  are  doing  it  for.  It's  a 
waste  of  money. 

Stable  user  needs 

Program 

Success 

4 

Funding  and  the  congress  looks  at  the  way 
we’re  performing.  You  know  you  can  get  all 
the  money  you  need  but  you  need  to  get  it  at 
the  correct  intervals  and  the  correct  amounts. 

Funding  time 

Program 

Success 

4 

And  you  have  to  be  trusted  by  the  congress  that 
you  are  spending  it  correctly.  If  you're  not  then 
they'll  take  it  away  and  you  won’t  succeed.  So, 

I  would  say  the  funding  the  way  you  get  is  very 
important. 

Funding 

Program 

Success 

4 

But  it  is  very  difficult  to  say  the  most,  they're 
all  so  important.  The  way  we  assess  our 
performance  and  the  way  we  receive  funding 
are  probably  the  biggest  thing  I  would  think. 

Funding 

Program 

Success 

4 

I  think  perfonning  well  and  having  a  good 
track  record  will  help  you  get  more  money. 
That's  also  very  important. 

Good  performance 
history 

Funding 

5 

I  am  trying  to  think  something.  There  are 
plenty  that  come  up  I  mean  you  can  look  at  on 
the  public  record.  For  example  GPS  program. 
You  know  it  started  up  very  well  and  they  did 
some  tremendous  work  because  they're  much 
focused  team,  they're  very  good  at  getting 
advocacy  for  the  program  maintaining  funding 
working  together  with  the  contractors  and 
picking  good  solutions. 

Establishing  teams 

Program 

Success 

5 

Then  is  the  program  group,  you  know  they 
made  initial  accomplishments,  they  launched 
all  these  satellites  and  then  you  know  now  it's 
the  next  contract  and  everything  is  running  into 
the  some  of  typical  acquisition  problems. 

There  is  a  lot  of  scrutiny;  they're  spending 
money  than  they  should  be.  They're  running 
into  problems  like  we  call  vanishing  vendors. 

Vanishing  vendors 

Challenges 
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5 

That's  a  major  problem  for  acquisition 
contracts.  Because  let’s  say  you  have  a 
company  that  makes  particular  items  that's 
vital  to  your  program.  If  that  company  goes 
away  then  you  have  to  develop  that  capability 
some  other  way.  In  this  case  it's  the  producer 
of  the  atomic  clock. 

Vanishing  vendors 

Challenges 

5 

Another  example  is  you  need  certain  batteries 
for  a  vehicle,  like  an  unmanned  vehicle  and  the 
company  that's  making  those  batteries  goes  out 
of  business.  Now  what?  What  are  we  doing? 
You  know  you  have  to  redesign  everything,  go 
to  a  different  type  of  battery  and  all  these  kind 
of  things.  Those  are  certainly  major  issue. 

Vanishing  vendors 

Challenges 

5 

You  know  we  go  to  war  and  we  got  a  nice 
funding  line  before  the  war  starts.  And  now 
sudden  need  for  the  war  so  we  lose  all  our 
money  for  the  program  because  they're  going 
to  war,  for  the  active  conflict.  Those  kinds  of 
things  happen  you  know,  fairly  often  in 
programs. 

Unexpected 

circumstances 

Challenges 

6 

We’ll  see  what  happens  with  the  X-5 1  but  it's 
been  very  well  run  program.  You  always  your 
share  of  things.  But  we’ll  find  out  in  a  couple 
of  years,  I  mean  a  few  months  whether  or  not 
successful.  It  was  a  very  good  effort,  the  team 
was  very  good  and  that's  a  very  good  example 
whether  or  not  it  perfonns.  Because  its 
research  might  not  work  properly  but 
everything  leading  up  this  point  has  been  very 
good. 

Establishing  teams 

Program 

Success 

6 

We  had  a  lot  of  good  advocacy  all  over 
government,  we  had  people  in  the 
administration  who  had  been  in  the  previous 
administrations  had  strong  supporters  of  the 
program. 

Advocacy 

Program 

Success 
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6 

The  major  thing  that  made  this  acquisition 
successful  is  communication.  The  way  we  set 
up  the  contract,  the  way  we  gage  the 
performance,  the  way  we  were  able  to 
manipulate  the  money  so  that  we  could,  you 
know  you  have  to  change  the  way  you  execute 
the  contract  in  order  to  best  use  the  money  so 
we  were  very  good  at  meeting  expenditure 
goals.  Because  the  way  we  have  to  do  business. 
Again  we  are  looking  at  performance,  its 
expenditures  that  congress  cares  about  so  we 
had  to  make  sure  that  we  were  satisfying  those 
goals. 

Communication 

Program 

Success 

6 

And  we  were  very  good  at  keeping  government 
involved  in  everything  that  was  happening 
because  as  long  as  refreshing  their  minds  they 
think  favorably  upon  the  program. 

Collaboration 

Program 

Success 

6 

You  have  reviews  you  go  through,  and  you 
have  to  please  all  kinds  of  different  people  to 
make  sure  that  they  don’t  think  your  program  is 
not  valuable.  So  we  were  very  good  at 
advocating  the  value  of  the  program  at 
speaking  the  entire  chief  scientist  of  the  Air 
force  for  example  was  a  great  advocate. 

Advocacy 

Program 

Success 

6 

You  know  breathing  congress,  breathing 
various  staffers,  these  kinds  of  things.  Playing 
the  politics  is  very  important  in  making  sure  it 
works. 

Politics 

Program 

Success 

6 

Y ou  know  because  of  all  that  this  great  relationship 
with  the  contractors,  and  having  knowledgeable 
people  on  staff,  working  with  great  other  and 
government  organizations,  getting  everyone 
involved.  The  more  people  getting  involved  like  an 
advisory  capacity  not  necessarily  in  an  execution 
but  to  look  at  it  better  is  because  all  their 
organizations  think  fluently  upon  it.  Y ou  have  to 
know  who  need  to  engage  what  the  requirements 
are.  You  need  to  talk  to  environmental  people;  you 
need  to  talk  to  different  offices.  If  you  keep  them 
early  and  you  know  everybody  is  happy  to  gain  the 
information  they  need  things  go  smoother  than  if 
you  forget  to  talk  to  somebody  and  later  they're  like 
"Oh,  you  didn't  talk  to  us."  You  know  we  are  going 
to  stop  you  and  you  come  into  bureaucracy. 

Collaboration 

Program 

Success 
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Proactivity.  That  was  the  major  thing  that  made 
that  program,  the  execution  work. 

Proactivity 

Program 

Success 

7 

I’d  say  number  one  that  you  get  the  vehicle  or 
the  item  that  you  wanted  to  get.  That  it  works 
the  way  it  is  supposed  to.  Those  are  big  ones. 

Product  Quality 

Program 

Success 

7 

You  need  to  satisfy  the  customer.  Otherwise 
there  is  no  point  to  doing  it.  If  you  just  get  a 
cool  item  nobody  wants  then  you  fail. 

Customer 

satisfaction 

Program 

Success 

7 

If  you  stayed  in  your  cost,  I  mean  as  long  as 
the  others  true  and  then  keeping  the  cost  is 
important. 

Cost 

Program 

Success 

7 

And  if  you  get  it  right  time,  you  don't  want  you 
know  to  develop  a  great  item  twenty  years  after 
be  useful  then  what?  What  did  you  do  it  for? 

Schedule 

Program 

Success 

7 

And  then  also  another  way  to  look  at  it.  It's  in 
the  future  of  that  item  is  easier  to  support  and 
maintain  then  the  overall  cost,  life-cycle  cost 
will  be  lower.  So  I  think  you  need  really  look 
at  to  sustainment  and  how  well  we  maintain, 
how  easiest  to  maintain  parts  and  knowledge 
about  the  system. 

Sustainment 

Program 

Success 

7 

I  mean  look  at  the  B-52.  It's  been  in  the 
inventory  more  than  fifty  years.  It  is  very 
successful  program  and  still  does  the  job  it  was 
designed  to  do.  So,  there  are  a  number  of 
programs  like  that  they  have  been  very 
successful  based  on  the  capability  they  offered 
for  the  price  and  schedule. 

Sustainment 

Program 

Success 

1 

Not  that  I’m  aware  of. 

2 

No,  I  haven’t. 

3 

Not  sure. 

4 

All  the  program  I’ve  worked  with  had  no 
funding  issues.  The  main  issue  was  to  meet 
schedule  for  the  sustainment  mod  program. 

Schedule 

Challenges 

5 

Can’t  think  of  one.  I  guess  I’ve  been  lucky  in  a 
way  as  I’ve  been  put  on  programs  that  were 
already  in  place  and  making  deliveries.  Of 
course,  that  takes  away  a  lot  of  the  potential 
learning  I  could  have  obtained  by  being  on 
“bad”  programs. 

Successful  program 

Program 

Success 
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I  was  not  the  program  manager,  however,  for 
years,  the  E-3  crewmembers  requested  new 
mission  crew  seats  for  the  planes  in  order  to 
reduce  crew  fatigue.  The  “old”  seats  in  place 
for  25  years  did  not  recline  at  all  which  is  tiring 
for  crewmembers  on  8+  hour  long  missions.  A 
fellow  program  manager  was  able  to  find 
funding  within  the  Air  Force  to  replace  the 
mission  crew  seats. 

Funding 

Program 

Success 

6 

These  seats  are  being  or  have  been  installed 
aboard  the  AWACS  fleet,  reducing  crew 
fatigue  and  improving  crew  morale. 

Customer 

satisfaction 

Program 

Success 

7 

If  my  leadership  and  the  users  are  satisfied 
with  the  program  and  the  program  stays  within 
legal  bounds. 

Customer 

satisfaction 

Program 

Success 

1 

We  did  a  rolling  plan.  Every  six  months  we  do 
a  plan  for  the  rest  of  the  term  we  had.  This 
considers  the  amount  of  money  that  we  are 
going  to  spent  on  manpower  and  materiel. 

Monthly  reviews 

Contract 

development 

1 

Lockheed  Martin  would  have  a  higher  price  for 
us,  we  couldn’t  do  that  and  we  shared  the  risk. 

Sharing  risk 

Contract 

development 

2 

We  had  a  very  good  experience  with  Martin 
Baker  company  from  France.  We  had 
something  similar  to  outsourcing.  They  were 
supplying  ejection  seats  for  the  aircraft. 

Different  Company 

Environment  of 
the  Contract 

3 

All  the  supply  chain  is  very  complicated  and 
integrated  so  it’s  very  difficult  to  apply  PBA.  It 
doesn’t  apply  so  easily.  We  have  several  type 
of  services  for  engines,  aircrafts  overhauling, 
textile,  ... 

Complex  contract 
environment 

Challenges 

4 

Failures  in  the  integrated  logistics  support.  The 
problem  was  tier  1,  tier  2  suppliers.  Sometimes 
some  of  them  were  disappeared.  Not  only  for 
the  electronics,  but  also  for  the  engines. 

Vanishing  vendors 

Challenges 

5 

Sometimes  you  are  not  fast  enough  to  review 
the  contract.  The  number  of  aircraft  that  we 
need  for  accomplishing  flying  plan  reduced 
dramatically.  We  were  not  fast  enough  to 
reduce  the  requirement  of  engines  that  we  have 
done  to  the  company. 

Flexibility 

Program 

Success 
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6 

Instead  of  paying  for  the  materials  that  we  are 
using  and  instead  of  paying  for  manpower 
hours  that  the  company  used,  we  will  pay  just 
and  only  for  the  milestones.  We  have  to  pay 
more  but  all  responsibility  and  risk  is  on  the 
side  of  the  company.  This  work  much  better. 

Sharing  risk 

Program 

Success 

6 

The  more  complete  is  a  system  the  easier  to  go 
to  PBL  or  to  have  a  one  single  measure, 
something  that  gives  you  an  overall  idea  of  the 
performance  of  the  supplier.  A  complete 
aircraft  is  easy  to  translate  the  risk  to  the 
company.  When  you  are  contracting 
maintenance  of  different  stuff,  different 
components  for  different  aircraft  this  is  much 
difficult  to  do. 

Variety  of  Service 

Challenges 

7 

When  you  got  what  you  expected  to  have. 

Customer 

satisfaction 

Program 

Success 

7 

First  the  number  of  deliveries,  then  the 
schedule  of  deliveries,  now  we  are  working  on 
quality  of  deliveries.  When  you  satisfy  the 
number  of  deliveries  then  you  focus  on  the 
schedule  of  deliveries.  After  these  you  focus  on 
the  quality  and  the  reliability  of  what  you  had. 

Number  of  delivery 

Program 

Success 

7 

It  depends  on  a  lot  of  factors.  But  first  is 
quantity,  then  is  time  and  after  that  is  quality. 

Quantity 

Program 

Success 

7 

We  have  a  lot  of  stuff  that  is  near 
obsolescence.  Sometimes  you  need  a  spare  and 
the  spare  is  not  available  on  the  shelf.  The 
company  has  to  rebuild  the  assembly  line 
again. 

Vanishing  vendors 

Challenges 

7 

In  some  case  PBA  is  wonderful.  For  example 
for  the  UN  mission  in  the  Cyprus.  Instead  of 
having  we  could  do  PBA  for  at  least  2 
helicopters.  That  having  to  take  care  of 
everything  for  just  two  helicopters  in  Cyprus.  I 
go  to  change  to  PBL  with  no  doubts  in  that 
case. 

Variety  of  Service 

Program 

Success 

1 

For  our  perfonnance  based  acquisitions  we  had 
quality  assurance  team  that  was  located  on 
each  side  where  our  products  were.  We  had 
quality  assurance  team  that  evaluated  the 
contractor  for  each  specific  location  and  then 
the  program  as  a  whole. 

Establishing  teams 

Program 

Success 
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1 

And  they  reported  back  to  us  what  had  been 
achieved  to  that  point  what  was  acceptable 
work  what  was  unacceptable  work,  to  pay  or 
not  to  pay  for  based  on  the  contractor  request 
for  payment. 

Periodic  reports 

Contract 

development 

1 

What  worked?  The  best  was  having  the  people 
on  location.  Because  they,  subject  matter 
experts,  could  see  the  daily  activities  by  the 
contractor.  You  could  see  what  the  contractor 
was  doing  constantly. 

Collaboration 

Program 

Success 

1 

When  you  are  dealing  with  software  and  some 
other  things  that  go  with  it,  it's  not  a  definite 
finish  date.  So  there  always  going  to  be 
upgrades  always  little  quirks  that  need  to  be 
worked  out.  And  sometimes  it  could  take 
longer  than  expected  because  something  fails 
in  the  quote  that  they're  writing. 

Type  of  job 

Challenges 

1 

So,  that's  why  PBA  works  in  this  case  because 
you're  getting  your  finished  product  for  the 
price  of  the  product  not  an  hourly  rate  for  the 
people  working  on  it. 

PBA  metrics 

Program 

Success 

1 

On  the  program  I  worked  on  yes  there  were 
cost  benefits.  Just  because  of  the  complex 
nature  of  what  they  were  working  on  if  we  had 
paid  them  in  hours  of  what  they  did  as  opposed 
to  completing  the  project  we  would  have  spent 
a  lot  more  money. 

PBA  metrics 

Program 

Success 

1 

Schedule.  Sometimes  they  had  incentive  to 
finish  faster.  Because  they  were  getting  paid 
just  based  on  the  products  so  they've  finished 
faster. 

Schedule 

Program 

Success 

2 

I  don’t  know  the  methods  that  they  typically 
use.  I  think  you  have  to  learn  about  one  the 
culture  of  the  country  that  they  are  coming 
from,  And  what  their  typical  business  practices 
are  to  determine  if  they  accept  performance 
based  strategy. 

Work  ethics 

Culture 

2 

It  is  important  to  get  very  educated  on  the  other 
countries  that  you  are  dealing  with  and  the 
contractors. 

Education 

Culture 

3 

It  is  a  long  term  contract;  they've  been  doing  it  that 
way  a  long  time  so  it  is  easy  to  continue  that. 
Because  the  contractor  is  very  familiar  with  it,  they 
are  accepting  the  way  you  do  business. 

Experienced 

contractor 

Program 

Success 
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3 

I  think  sometimes  you  have  to  look  at  what  the 
product  is  you  are  trying  to  get  out  of  it.  If  it's 
something  that  you  can  evaluate  performance 
and  evaluates  completion  levels  than  I  think 
PBA  is  best  way  to  go. 

PBA  metrics 

Program 

Success 

3 

Having  a  non-US  contractor  they  have 
experienced  of  what  is  needed  in  that  location 
and  what  type  of  methods  they  need  to  use  for 
construction  and  things  like  that.  The 
advantage  is  definitely  to  the  non-US 
contractor  in  that  situation. 

Experienced 

contractor 

Program 

Success 

3 

It  is  advantage  that  if  you  are  looking  for 
something  local.  So  if  you  are  trying  to  do 
work  in  Turkey  you  probably  want  a  Turkish 
contractor.  Because  they  have  local  knowledge 
they  know  what  the  culture  is. 

Experienced 

contractor 

Program 

Success 

4 

I  think  requirements.  I  think  having  very  good 
understanding  of  what  the  government  wants  to 
get  out  of  the  program  and  what  the  contractor 
is  going  to  provide 

Defining 

Requirements 

Program 

Success 

4 

And  making  sure  that  regardless  of  whom  the 
contractor  is  and  what  they're  doing  for  you 
that  they  have  the  same  interpretation  of  what 
you  want  as  you  do. 

Communication 

Program 

Success 

4 

And  that  the  requirements  are  set  and  not 
constantly  changing.  If  the  requirements 
change,  they  fluctuate  then  the  likelihood  you'll 
get  what  you  want  in  the  end  goes  down 
dramatically. 

Defining 

Requirements 

Program 

Success 

4 

The  language,  no  matter  what  language  it  is,  is 
very  subjective.  You  can  look  at  the  same 
words  as  I  look  at  and  get  a  totally  different 
interpretation  of  what  that  means.  So,  language 
creates  a  big  obstacle. 

Language 

Challenges 

4 

The  written  word  is  based  on  their  knowledge 
and  their  experience.  So,  everyone  has  different 
experiences,  the  contractor  is  coming  from  the 
contractor  side  trying  to  figure  out  what  it  is  you 
want  and  you  are  coming  from  the  government 
side  trying  to  put  into  words  what  you  have 
pictured  in  your  head.  So  it  doesn't  matter  to 
speak  the  same  language  naturally  or  not  there 
can  be  a  lot  of  conflicts  interpreting  the  language. 

Interpretation  of 
contract 

Challenges 
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5 

I  found  contractors  like  to  paint  a  pretty  rosy 
picture.  If  they  give  you  bad  news  they  feel 
they'll  be  evaluated  horribly  and  the  contract 
could  end. 

PBA  metrics 

Contract 

development 

6 

Having  a  contractor  that  was  really  willing  to 
do  what  we  needed  and  mould  the 
requirements  and  change  a  few  things  the  last 
minute  allowed  us  to  speed  up  the  production 
line. 

Flexibility 

Program 

Success 

7 

When  you  meet  the  requirements  of  the 
contract,  when  you  meet  the  original  intent  of 
your  design  of  your  product  and  it's  been 
successfully  used  without  major  failures  I  think 
that  is  a  success. 

Customer 

satisfaction 

Program 

Success 

7 

The  program  as  a  whole  can  be  a  success  even 
if  it  has  little  failures  along  the  way.  So,  you 
may  not  get  everything  done  right  on  the 
schedule,  or  you  may  have  major  cost  overrun 
that  you  have  to  make  up  in  the  overrun  but  if 
you  get  your  product  where  it  needs  to  be  used, 
who  it  needs  to  be  by  when  they  need  it  then  it 
is  a  success. 

Product  Quality 

Program 

Success 

1 

As  mentioned,  I  was  involved  with  a  cost-plus 
award  fee  type  contract.  The  award-fee 
process  required  a  quarterly  review  of  all 
award-fee  comments. 

Award  Fee 

PBA  metrics 

1 

Valid  comments  could  be  either  positive  or 
negative  and  ultimately  determined  whether 
the  contractor  made  the  award-fee  bonus. 

Award  Fee 

PBA  metrics 

1 

Comments  were  received  from  various 
government  units  involved  with  the 
contractor’s  work. 

Various  comments 

PBA  metrics 

1 

In  my  mind  there  wasn’t  enough  rigor  in  the 
award  fee  process.  Often  the  contractor  would 
receive  praise  for  what  seemed  to  me  to  be  part 
of  their  normal  responsibilities. 

Vague  awarding 

Challenges 

1 

For  those  collecting  the  award-fee  comments, 
it  was  always  easier  to  find  and  report  the  good 
things  over  the  bad. 

Award  Fee 

PBA  metrics 

1 

In  the  end,  I  don’t  think  the  contractor  ever 
really  went  above  and  beyond  what  they  were 
originally  paid  to  do. 

Payment 

Program 

Success 
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1 

Basically,  it  didn’t  result  in  CSP  benefits 
because  they  already  had  their  money.  Which 
is  fine,  except  they  probably  received  financial 
award  based  on  the  assumption  that  they  did  go 
above  and  beyond. 

Payment 

Program 

Success 

2 

I  have  never  been  involved  with  non-U. S. 
contracts,  so  unfortunately,  I  am  unable  to 
comment. 

Not  involved 

Environment  of 
the  Contract 

3 

This  type  of  Acquisition  method  should  be 
used  in  situations  where  the  government  can 
see  tangible  benefits  from  the  contractor’s 
outstanding  performance. 

Tangible  benefit 

Environment  of 
the  Contract 

3 

In  my  case,  it  didn’t  seem  to  be  the  most 
effective  method  in  the  operations  and 
sustainment  phase. 

Sustainment 

Program 

Success 

3 

So  to  answer  your  question,  I  would  be  most 
likely  to  use  it  in  an  early  acquisition  phase, 
where  schedule  and  cost  are  most  vulnerable. 

Cost  and  Schedule 

Challenges 

4 

I  think  the  biggest  obstacle  to  success  is  the 
instability  of  the  user  requirements  and 
schedule  expertise,  both  of  which  lead  to  cost 
overruns  that  you  can  never  recapture. 

Instable  user 
requirements 

Challenges 

4 

One  of  the  reasons  for  requirements  instability 
is  that  Program  Managers  do  not  have  the  right 
amount  of  authority  to  direct  their  programs. 

Authority 

Challenges 

5 

A  request  for  proposal  was  sent  out  in  order  to 
purchase  around  700  high  quality  surveillance 
cameras  for  a  perimeter  detection  system. 
Proposals  were  received  and  the  contract  was 
awarded  to  a  small  business  that  essentially 
won  because  they  gave  the  lowest  bid. 

Lowest  bid 

Type  of 
Contract 

5 

After  they  won  the  bid,  they  tried  to  buy  the 
cameras  on  credit  from  the  manufacturer,  but 
were  unsuccessful  for  several  months  because 
they  only  had  around  10K  of  capital  assets. 

Funding 

Challenges 

5 

After  the  first  batch  of  cameras  was  delivered, 
it  was  noted  that  they  were  not  equipped  with 
essential  cabling  required  to  deliver  the  video 
and  data  streams  from  the  cameras  to  the 
monitoring  stations. 

Product  Quality 

Program 

Success 
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5 

This  cabling  was  clearly  described  in  the  RFP, 
but,  as  it  turned  out,  the  small  business  did  not 
understand  the  requirement  and  thusly  did  not 
include  it  in  their  original  proposal.  That  is 
how  they  were  able  to  outbid  the  competition. 

Communication 

Program 

Success 

5 

At  this  point  the  Contracting  Officer  would 
only  allow  the  repurchase  of  the  cabling 
through  the  same  small  business  and  only  with 
the  same  source  of  money  that  had  paid  for  the 
original  contract. 

Communication 

Program 

Success 

5 

This  turned  out  to  be  a  disaster  because  the 
original  money  had  come  from  several  Air 
Force,  Anny,  and  Marines  operational  units 
who  paid  for  their  portions  of  the  systems 
largely  with  one-time  money. 

Funding 

Challenges 

5 

The  Project  Manager  ended  up  having  to  go 
around  to  all  these  units  and  ask  for  the 
additional  money  to  pay  for  the  cabling  that  the 
units  already  thought  they  had  paid  for.  What 
a  disaster! 

Funding 

Challenges 

6 

Honestly,  hero  stories  are  harder  to  come  up 
with.  It  seems  like  every  acquisition  program 
encounters  problems  somewhere  along  the  line. 

No  perfect  program 

Program 

Success 

7 

Acquisition  success  metrics  are  extremely 
difficult  to  measure. 

Difficult  to  measure 

PBA  metrics 

7 

It  basically  comes  down  to  cost,  schedule,  and 
performance. 

Cost,  schedule  and 
performance 

Program 

Success 

7 

I  think  perfonnance  should  ultimately  be  the 
determining  factor. 

Performance 

Program 

Success 

7 

If  the  system  can  perform  the  way  it  was 
originally  intended  to,  then  that  is  success. 

Perfonnance 

Program 

Success 

7 

Often  times  these  days,  cost  is  metric  of 
success  because  of  the  amount  of  attention  paid 
to  the  big  acquisition  programs  and  their 
noteworthy  cost  overruns. 

Cost 

Program 

Success 

1 

Well,  we  met  as  a  team,  and  developed  service 
summaries,  what  the  remedies  or  if  the 
contractor  doesn't  follow  through  in  doing 
something  how  we’re  going  to  handle  that. 

Planning  contract 

Contract 

development 

1 

We  did  quality  assurance;  training  so  that  the 
folks  who  were  overseeing  the  quality  of  the 
contract  would  understand  what  it  is  we're 
looking  for  in  that  contract. 

Planning  contract 

Contract 

development 
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1 

Everything  worked  fine.  If  we  had  something 
that  we  need  to  revise  then  we  revisited. 

Contract  success 

Contract 

development 

1 

We  had  challenges  of  trying  to  develop  so  the 
summaries  and  how  to  come  up  with  remedies 
for  what  happened  and  what  we  should  revise 
from  the  contractor  if  that  didn’t  work. 

Revision  of  the 
contract 

Challenges 

1 

If  some  part  of  that  didn’t  work,  something  in 
the  performance  was  deficient  how  we  were 
addressed  that.  So,  those  were  the  challenges 
of  how  to  identify  remedies. 

Performance 

deficiency 

Challenges 

1 

I  see  a  lot  of  overhead  and  performance  based 
acquisition  and  not  that's  bad  but  it  takes  a  lot 
of  time.  I  can't  answer  that  whether  PBA  save 
us  money. 

Cost 

Challenges 

2 

Well,  it's  the  same  as  what  I've  told  you, 
because  what  I've  been  talking  to  you  about  is 
an  FMS  contract,  so  foreign  military  sales 
contract.  Is  just  that  DOD  Air  Force  who  put 
the  contract  between  us  and  the  contractor, 
who  is  also  American  firm  that  is  for  a  foreign 
government  and  for  support  of  a  foreign 
government?  So  saying,  our  saying,  rules  apply 
whether  it's  in  the  States  versus  whether  it  is 
outside. 

US  or  non-US 
contract 

Environment  of 
the  Contract 

2 

You  know  that's  when  you  say  F-15,  F-16  and 
those  I  don’t  know  how,  our  guess  is  the 
maintenance  contract  but  I  have  support 
service  contract.  It's  not  for  a  system;  it's  for 
services,  to  people. 

Service  contract 

Type  of 
Contract 

2 

I  am  not  done  a  PBA  other  than  foreign 
military  sales. 

FMS  contract 

Environment  of 
the  Contract 

3 

It  means  that  they  were  paid  for  foreign 
military  money,  non-US  money,  but  the 
contract  is  between  the  US  government  and  US 
contractor. 

Non-US  money 

Funding 

3 

No  not  that  I  know.  Not  in  my  near  future  I 
know  we  don’t  plan  to  implement  any  contract 
with  the  non-US  contractors. 

Non-US  Contractor 

Environment  of 
the  Contract 

3 

Because  as  US  we  typically  are  going  to  follow 
the  same  guidelines  that  we  have  for  with  US 
contractors. 

Non-US  Contractor 

Challenges 
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3 

I  mean  there  would  be  a  few  things  that  maybe 
wouldn’t  be  applicable  but  generally  US 
entering  into  a  contract  we  have  certain 
guidelines  that  we  follow  so  they  would  unless 
the  FAR  says  it  doesn’t  apply  to  than  it  would 
apply. 

Contracting 

guidelines 

Environment  of 
the  Contract 

4 

Funding. 

Funding 

Challenges 

4 

The  most  difficult  part  is  to  find  someone 
qualified  to  be  willing  to  manage  the  contract, 
the  performance. 

Finding  Contractor 

Challenges 

4 

To  oversee  the  performance  and  to  do  periodic 
reviews  to  ensure  that  the  contractors 
performing  in  accordance  with  contract. 

4 

So,  I’d  say  the  most  challenging  part  is  to  find 
someone  to  do  that.  Especially  since  our 
contractor  is  working  out  of  country. 

Non-US  Contractor 

Challenges 

5 

No  horror  story. 

6 

No  hero  stories.  Everything  went  well. 

6 

It's  the  one  according  to  the  way  we  expected. 
This  is,  this  contract  is  being  an  existence  for  a 
year.  So,  you  know  and  the  same  contractor 
pretty  much  knows  what  they're  doing. 

Qualified  contractor 

Program 

Success 

6 

So,  we  don’t  have  the  issues  some  contracts 
had. 

Routine  contract 

Program 

Success 

7 

Well,  of  course  time,  and  the  service  that  we’re 
receiving  quality  service,  when  we  couldn’t 
recognize  that  we  were  receiving  timely  and  up 
to  quality  that  we  expect  it's  a  successful 
contract,  our  customer's  satisfied. 

Time,  Customer 
Satisfaction 

Program 

Success 

7 

I  think  satisfaction  has  to  come  in  first. 

Customer 

Satisfaction 

Program 

Success 

7 

Because  if  our  customer  is  satisfied  then  we 
won’t  have  needed  to  do,  we  may  not  get  the 
opportunity  to  do  that  contract  again. 

Customer 

Satisfaction 

Program 

Success 

7 

We  have  to  satisfy  our  customer  first  and  then 
you  can  say  cost. 

Customer 

Satisfaction 

Program 

Success 

1 

Basically  in  structuring  the  contract  they 
wanted  us  to  create  top  level  kind  of  short 
summary  what  we  were  trying  to  accomplish 
with  the  contract  and  you  start  to  break  it  down 
into  what  were  the  performance  measures  that 
would  help  us  achieve  success. 

Performance 

measures 

Contract 

development 
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1 

Define  the  task  and  create  a  level  of  measure 
effectiveness  or  level  of  performance  that  we 
wanted  the  contractor  achieve.  The  challenging 
part  was  to  create  some  measure  of 
effectiveness  for  an  engineering  task  that  made 
sense  that  was  clearly  definable  and  could  be 
used  effectively. 

Defining 

Requirements 

Contract 

development 

1 

The  challenge  was  trying  to  get  the  engineers 
in  our  office  and  program  managers  and 
everybody  sit  down  and  figure  out  what  is  a 
reasonable  measure  to  determine  that  they've 
done  effectively  and  we’re  getting  the  product 
we  want. 

Defining 

performance 

metrics 

Contract 

development 

1 

You  have  to  sit  down  and  define  upfront  and 
get  both  parties  to  agree  here  is  what  our 
measures  of  quality  on  the  final  product  are. 
Especially  when  you  are  doing  something  very 
intellectual  based  work.  That  can  be  difficult  to 
create  a  much  defined  case  easy  measurable 
list  of  items. 

Defining 

performance 

metrics 

Challenges 

1 

I  think  one  of  the  things  that  weren’t  done  well 
when  we  transitioned  was  the  level  of  training 
and  education  wasn’t  necessarily  where  it  could 
have  been. 

Training 

PBA  metrics 

2 

As  far  as  the  idea  of  being  able  to  use 
performance  based  measures  on  a  non-US 
contractor  I  don’t  see  any  reason  why  that 
wouldn’t  work  it  all.  Both  parties  would  still 
have  to  agree  on  the  measures  of  the 
performance  and  the  effectiveness  but  the 
principle  of  perfonnance  based  acquisition  I 
don’t  see  why  that  would  change  it  all  based  on 
the  nationality  of  the  contractor  or  the 
government. 

No  difference 

Environment  of 
the  Contract 

3 

If  I  had  the  opportunity  to  create  a  contract 
with  a  non-US  contractor  I  would  expect  that 
still  work. 

No  difference 

Environment  of 
the  Contract 

3 

For  us  working  with  a  US  contractor  just  for 
security  concerns  and  some  of  our  regulatory 
issues  it  is  easier.  As  far  as  if  we  work  with  a 
contractor  outside  the  US  there  are  concerns 
about  the  security  and  proprietary  data  and  stuff 
like  that  need  to  be  handled  a  little  bit  differently. 

Security 

Environment  of 
the  Contract 
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Dealing  with  a  new  contractor,  a  foreign 
contractor  I  would  probably  b  more 
comfortable  using  performance  based 
acquisition  because  now  between  government 
and  the  contractor  we  agreed  upon  levels  of 
performance  so  that  everybody  knows  upfront 
what  is  expected  of  each  party.  That  would 
give  me  a  level  of  comfort. 

No  difference 

Environment  of 
the  Contract 

4 

Getting  the  requirements  and  the  scope  of  the 
project  defined  upfront  correctly.  We  had  a  lot 
of  issues  later  on  the  program  where  we  have 
thought  through  everything  upfront  so  we’d 
find  that  realized  later  on  way  we've  got 
additional  work  or  we  need  to  change  this. 

Defining 

Requirements 

Challenges 

4 

Because  not  only  do  you  have  to  think  about 
what  tasks  need  to  be  accomplished  but  which 
are  critical  tasks  you  need  to  have  done  and 
what's  a  measure  of  effectiveness  that  you 
needed  done  to.  In  order  to  create  to 
performance  based  contract  there  is  actually 
more  work  more  thinking  that  has  to  be  done 
upfront  to  releasing  the  contract. 

Defining 

performance 

metrics 

Contract 

development 

4 

If  you  got  a  contractor  that  historically  you  had 
issues  in  a  particular  area  spending  the  time 
upfront  to  come  to  an  agreement  of  what  is 
expected  and  document  those  expectations  and 
what  is  going  to  be  done  to  hold  them 
accountable  is  probably  going  to  pay  off  and 
saving  you  for  problem  down  the  road. 

Experienced 

contractor 

Program 

Success 

5 

This  individual  got  the  idea  of  the  concept  and 
went  to  Air  force  chief  of  staff  at  the  time  and 
told  him  about  it.  Chief  of  staff  told  him  it  is 
brilliant  idea  here  is  the  budget,  go  make  it 
happen.  The  guy  didn't  talk  to  A- 10  program 
office,  didn’t  talk  to  pilots.  When  it  came  to 
program  office  the  pilots  said,  I  don’t  want 
them  in  my  cockpit. 

Establishing  teams 

Program 

Success 

5 

In  the  end  it  never  went  on  the  aircraft.  The  test 
community  tried  flying  with  the  configuration 
with  it  and  detennines  that.  It  was  a  safety 
issue  because  when  they  move  the  stick  it 
would  hit  this  item,  it'll  block  the  stick  from 
full  movement. 

Defining 

Requirements 

Program 

Success 
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Because  it  was  driven  from  high  level  nobody 
was  going  to  give  up  even  when  the  data  was 
already  there  saying  it  is  not  practical  solution. 
Used  up  a  lot  of  resources,  money,  and  man 
power  trying  to  do  that  in  the  aircraft. 

High  level  pressure 

Challenges 

6 

I  came  on  to  the  program  and  started  to  do 
looking  at  it  and  basically  came  into 
determination  there  was  no  way  it  was  going  to 
cost  them,  they  thought  it  was  going  to  cost 
much  more,  and  they  made  a  lot  of 
assumptions  on  how  quickly  things  could  be 
done,  and  they  also  missed  a  lot  of  activities 
that  needed  to  be  done. 

Communication 

Program 

Success 

7 

It  needs  to  meet  the  user’s  needs  and 
expectations.  Performance  of  the  system  and 
the  environment  where  the  user  uses  it. 

Customer 

satisfaction 

Program 

Success 

7 

Also  meeting  the  cost  and  schedule  targets  are 
set  to  the  baseline  of  the  program  being  able  to 
stay  within  that. 

Cost,  schedule 

Program 

Success 

7 

Being  able  to  turn  around  and  educate,  prepare 
the  workforce  for  future  programs  by  executing 
something  successfully.  Prep  for  the  future, 
prep  for  the  workforce  being  able  to  continue 
doing  that  work. 

Training 

Program 

Success 

1 

Have  you  ever  been  involved  in  a  contract 
using  performance  based  acquisition  methods? 
Yes 

1 

If  so,  can  you  describe  the  methods  that  were 
used?  Award  Fees  for  contractors,  working  in 
teams  to  establish  a  schedule,  Integrated 
Working  Groups 

Establishing  teams 

PBA  methods 

1 

What  “worked”?  Award  Fees  -  Contractors 
worked  hard  to  reach  the  maximum  incentives 

Award  Fee 

Program 

Success 

1 

What  did  not  “work”?  Working  in  teams  to 
establish  schedule  -  there  was  never  any 
agreement  between  parties  resulting  in  endless 
meetings  re-base  lining  the  schedule.  The  IPT 
was  simply  a  gripe  session. 

Schedule 

Program 

Success 

1 

In  your  estimation,  did  performance  based 
methods  result  in  cost,  schedule,  or 
performance  benefits  compared  to  other 
methods?  No 

Overall  PBA  is 
negative 

Program 

Success 
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Have  you  been  involved  with  non-U. S. 
contracts?  NO 

3 

In  the  future,  do  you  plan  to  implement 
performance  based  acquisition  methods  (in  a 
U.S.  environment  or  in  a  non-U. S. 
environment)?  Not  sure  - 1  am  going  to  a  new 
unit  and  I  will  have  to  see  what  measures  they 
have  in  place. 

Overall  PBA  is 
negative 

Program 

Success 

4 

In  acquisition  programs  in  which  you  have 
been  involved,  what  has  been  the  most  critical 
issue  or  obstacle  to  success  (e.g.  cost, 
sustainability,  meeting  the  user’s  requirements, 
schedule,  etc.)?  Cost  and  Schedule 

Cost  and  Schedule 

Program 

Success 

5 

Can  you  share  an  acquisition  “horror  story” 
with  me,  when  something  went  wrong?  Yes 
There  I  was.  I  just  became  the  Deficiency 
Reporting  and  Investigation  System  (DRIS) 
program  manager.  The  system  was  on  its  way 
to  shut  down  due  because  a  new  system  was 
coming.  However,  no  one  wanted  to  actually 
pull  the  plug  because  they  knew  it  would  be 
their  job.  Therefore,  every  time  I  tried  to  get 
anything  done,  it  was  met  with  opposition. 

Also,  my  division  was  under  one  chain  of 
command  and  the  users  of  the  system  (the 
customer)  were  under  another.  I  tried  to  write 
an  agreement  between  everyone  specifying  the 
funding  and  shut  down  schedule.  It  got  road 
blocked  so  many  times  that  it  never  got  signed 
off.  This  was  an  important  document  too.  It 
needed  to  be  signed  and  everyone  knew  it.  I 
wound  up  moving  to  AFIT  without  it  done. 
After  a  year,  I  saw  the  current  program 
manager.  They  are  still  operating  under  the  old 
agreement  because  no  one  ever  spent  the  time 
to  finish  getting  the  new  one  signed.  I'm  not 
sure  if  PBA  would  have  done  any  good.  The 
story  does  show  insight  into  the  mentality  of 
some  acquisition  workers.  Change  is  difficult 
to  attain. 

Resistance  to 
Change 

Challenges 
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6 

Can  you  share  an  acquisition  “hero  story”  with 
me,  when  something  went  right? 

I  was  the  Executive  Officer  for  the  554th  when 
a  new  Commander  arrived.  He  took  many  of 
the  programs  and  started  holding  people 
accountable.  Also,  if  people  didn’t  live  up  to 
his  expectations,  they  were  removed  from 
leadership.  He  set  the  goals  which  were 
challenging  yet  attainable.  He  set  the 
consequences  and  the  rewards  and  people 
responded.  As  a  result,  2  programs  were  saved 
and  the  entire  group  returned  to  schedule. 

Setting  Rules 

Program 

Success 

7 

When  can  you  describe  an  acquisition  as 
successful?  When  something  is  actually  fielded 
and  works  as  specified 

Product  Quality 

Program 

Success 

7 

What  are  the  most  important  metrics  for 
determining  success?  usability,  cost,  schedule 

Usability,  Cost, 
Schedule 

Program 

Success 

1 

Successful  performance  based  contracting  is 
contingent  upon  a  relationship  with  a  capable 
and  competent  partner  that  is  willing  to  share 
risk. 

Sharing  risk 

Program 

Success 

1 

It  also  depends  on  the  cost  and  payment 
structure  of  the  contract  (i.e.  firm  fixed  price  vs 
cost  reimbursable)  plus  the  incentive  schedule 
meeting  and/or  exceeding  the  performance 
standards  in  the  contract. 

Cost 

Program 

Success 

1 

The  difficulty  lies  in  appropriately  defining  the 
performance  standards  to  ensure  that  the 
customer  gets  what  he  needs  (within  costed 
guidelines),  and  the  contractors  able  to  achieve 
those  standards  (within  the  resource 
restrictions  applied)  while  still  making  fair 
profit. 

Defining 

performance 

metrics 

Challenges 

2 

Not  sure  what  is  being  asked.  Our  contracting 
methods  are  essentially  the  same. 

No  difference 

Environment  of 
the  Contract 

3 

The  decision  will  not  be  mine,  but  this  seems 
to  be  the  way  of  the  future. 

Future  contracting 
method 

Environment  of 
the  Contract 

3 

But  it  is  easier  said  than  done,  since 
institutionally  we  lack  the  experience  and 
expertise  of  the  private  sector. 

Lack  of  experience 

Challenges 
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Performance  based  contracting  is  a  relatively 
new  concept  (especially  for  the  military  -  since 
we  are  used  to  "doing"  as  opposed  to 
"managing")  and  is  difficult  to  implement 
without  appropriate  training  (particularly  with 
respect  to  a  quality  management  system  on  the 
part  of  those  tasked  with  ensuring  that  the 
standards  in  the  contract  are  being  met,  as  well 
as  for  those  performing  financial  audit  and 
certification). 

Training 

Program 

Success 

4 

As  well,  there  needs  to  be  a  common 
understanding  (with  the  contractor)  of  how 
things  will  work.  It  all  comes  to  “expectation 
management". 

Communication 

Program 

Success 

5 

Not  really.  Most  problems  were  overcome  with 
appropriate  authorities  making  timely  and 
appropriate  decisions. 

Timely  decisions 

Program 

Success 

5 

You  need  to  identify  those  authorities,  and 
delegate  decision-making  to  the  lowest 
possible  level. 

Communication 

Program 

Success 

6 

- 

No  hero  story. 

Program 

Success 

7 

It  really  depends  on  the  type  of  procurement. 

Different 
procurement  type 

Program 

Success 

7 

In  the  case  of  logistics  support  to  deployed 
operations,  it  would  be  no  loss  in  operational 
effectiveness  due  to  alternative  service  delivery 
(ASD).  In  Canada's  case,  it  was  enhancing  our 
ability  to  deploy  forces  by  allowing  functions 
that  could  be  given  to  the  private  sector  to  be 
out  sourced. 

Alternative  service 
delivery 

Type  of 
Contract 

1 

Yes.  In  fact  the  one  we  use  currently  for  the 
contract  we  have  to  modify  your  aircraft  that 
Turkish  fleet  aircraft  is  a  format  performance 
based  acquisitions.  It's  a  sole  source  contract  so 
wasn’t  competitive  since  Lockheed  Martin  is 
the  sole  provider  for  the  F-16  aircraft. 

Sole  Source 

Type  of 
Contract 

1 

But  contracting  we  do  use  the  basic  tenets  of 
performance  based  acquisitions.  We  submit  a 
sue  contractor  comes  back  with  a  SOA,  with  a 
systems  requirements  document 
recommendation  and  that's  how  we  contract. 

Statement  of  Work 

Contract 

development 
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Well,  it's  the,  we  first  submitted  to  Lockheed 
and  our  proposal  was  statement  of  objectives. 
We  don’t  stay  the  specific  requirements,  you 
know,  down  to  the  very  level  of  detail. 

Statement  of 
Objectives 

Contract 

development 

1 

We  state  an  overall  goal  basically  objectives 
such  as  upgrade  the  avionics  similar  to  a 
certain  level. 

Stating  goal 

Contract 

development 

1 

And  then  through  meetings  with  the  contractor 
they  make  sure  they  understand  what  our 
requirements,  what  we’re  trying  to  do,  what 
mission  we  are  trying  to  meet. 

Meetings 

Contract 

development 

1 

So,  that  gives  the  contractor  flexibility  then  to 
go  in  or  we  don't  dictate  every  single  thing  that 
they're  to  do  we  give  them  latitude,  and  this  to 
work  you  have  to  have  a  contractor  that's  pretty 
competent  has  a  lot  of  experience  in  doing 
these  things. 

Flexibility 

Contract 

development 

1 

But  they  in  turn  will  break  that  statement  of 
objectives  more  into  solid  statement  we  call  it 
statement  of  work.  And,  it  will  define  the 
specific  task  that  the  contractor  will  do. 

Statement  of  Work 

Contract 

development 

1 

Also  to  guide  that  we  have  systems 
requirements  document.  And  that  document 
again  it  doesn’t  go  to  a  fine  degree  of  how  to  do 
the  task.  It's  more  what  we  need  accomplish  by 
the  end  of  the  task. 

Systems 

requirement 

document 

Contract 

development 

1 

There  are  mil-standards  and  certain  guidelines 
that  are  in  that  systems  requirements 
documents  and  that  has  a  lot  of  lower  level 
requirements  like  to  meet  some  electrical 
qualification  requirements. 

Systems 

requirement 

document 

Contract 

development 

1 

Also  we’ll  develop  Interface  Control 

Documents  (ICDs.  It  defines  the  interface 
between  various  systems  within  the  overall 
system. 

Interface  control 
documents 

Contract 

development 

1 

So,  by  using  those  documents  you  define  more 
technical  aspects  of  the  acquisition. 

Technical  Aspects 

Contract 

development 
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And  then  that's  how  we,  we  have  various 
design  reviews  and  we  see.  The  contractor  is 
basically  stage.  You  know  here  is  how  we  are 
going  to  accomplish  this  and  they  generate  the 
mini  plans  or  performance  measurements  that 
are  used  and  we  evaluate  those  success.  How 
do  we  know  we  achieve  success  by  using  those 
goals  or  the  documents  of  they  said. 

Design  reviews 

Contract 

development 

1 

So  in  plus  during  all  the  many  test  and  design 
reviews  and  evaluations  that's  how  you,  you 
know  you  have  to  have  some  way  to  know 
when  you  achieved  your  goal. 

Testing 

Program 

Success 

1 

I  think  overall,  the  general  process  works.  You 
have  some  circumstances  possibly;  I  can’t  say 
this  point  in  this  particular  effort  that  anything 
hasn't  worked.  You  face  problems  with 
individual  development  or  systems  that  are  in 
development  and  you  may  face  some  technical 
difficulties.  A  video  card  may  fail  because  the 
design  is  not  proper.  So  you  don't  get  the  right 
display.  Maybe  has  noise  or  something  so  you 
have  to  go  back  and  have  to  redesign  that.  Of 
course  that  causes  problems  downstream  from 
the  schedule  and  the  cost  impact.  But  the 
process  itself  you  know  we  were  able  to 
discover  the  issue  or  whatever  that's  been 
successful. 

Technical 

Difficulties 

Challenges 

1 

I  can't  say  that  there  hasn't  been  anything  that 
hasn't  work  that  I  would  say  fault  of 
performance  based  acquisition.  There  have 
been  failures  to  meet  technical  goals  or 
requirements.  Maybe  there  is  an  interface 
problem  between  systems.  But  that’s  more  a 
technical  issue  it's  not  really  performance 
based  acquisition  issue. 

Interface  Problems 

Challenges 

1 

I  guess  there  are  other  contracts  that  we  use  that 
use  performance  based  acquisitions  and  in  those 
in  some  circumstances  I  would  say  if  you  don't,  I 
am  talking  in  general  here  ok,  properly  define 
your  measurements.  Sometimes  they  can  be  done 
just  to  fill  a  square.  I  mean  to  say  'hey  we  are 
doing  performance  based  acquisitions'  and  you 
make  up  some  criteria  as  you  go  through  that 
really  don't  give  any  benefit  to  the  acquisitions. 

Defining 

performance 

metrics 

Challenges 
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It's  more  along  the  lines  of  'ok  we  have  to  do 
performance  based  acquisitions.  So  we  are 
going  to  establish  these  measurements.  And 
they  may  not  be  appropriate  for  the  acquisition. 
And  that's  the  key  to  get  measurements  that 
you  know  what  you  are  doing  is  correct. 

Defining 

performance 

metrics 

Program 

Success 

1 

If  you  are  going  a  contract  for  someone  to  do  a 
level  of  effort  testing  for  you  said  part  of  what 
you  want  it  to  do  test  our  measure  the  success 
of  that  buy  not  so  much  that  it  met  the  test  but 
it  did  it  within  certain  budget.  It's  a  level  of 
effort  you  are  not  sure  what  that  cost  is  but  you 
set  a  range.  If  you  don’t  really  if  the  test  results 
in  the  reaching  success  is  more  important  than 
a  certain  level  of  cost  you  may  succeed  in  your 
testing  but  because  of  your  cost  level  is  too 
high  you  really  say  "oh  we  haven't  met  our 
paragoals  or  we  exceeded  our  goals  in  cost  " 

Budget  Constraint 

Challenges 

1 

So  you  need  to  make  sure  what  you  are  doing 
really  measures  what  it  is  you  are  trying  to 
achieve  and  you  don't  artificially  create  some 
kind  of  goal  or  some  kind  of  measurements  for 
your  in  results  that  really  doesn’t  give  you  any 
benefit.  You  know  it's  not  relevant  to  what 
your  final  real  objective  is.  Because  the  goal  is 
you  have  a  mission  and  you  want  to  reach  that 
mission  and  you  need  to  make  sure  that  what 
you’re  contracting  for  is  in  line  with  what  your 
mission  goal  is. 

Defining 

performance 

metrics 

Program 

Success 

1 

So  if  it’s  ultimately  delivered  a  fighter  aircraft 
and  you  want  it  by  this  date  and  it  need  to  be 
and  obviously  cost  is  an  issue  you  need  to 
structure  the  program  so  that  you  can  measure 
those  things.  But  does  it  mean  you  failed  say  it 
you  have  to  make  tradeoffs  as  you  go  through. 

Cost  -  Quality  trade 
off 

Program 

Success 

1 

But  you  want  to  make  sure  that  those 
measurements  help  guide  you  to  reach  those 
goals.  If  it's  not  relevant  I  mean  if  whatever 
your  other  measurements  might  be  if  they  are 
not  relevant  to  that  in  result  you  have  kind  of 
wasted  both  the  contractor  time  to  track 
whatever  it  is  and  your  time  to  trying 
observing. 

Defining 

performance 

metrics 

Program 

Success 
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I  don’t  see  that,  that  has  been  really  influence 
so  much  by  the  performance  based  acquisition 
in  not  achieving  those  standards.  I  guess  I  say 
yeah  it  has  helped.  It  helps  like  I  said  if  you 
structured  it  right  it  helps  you  to  focus  on  the 
most  important  aspects  of  the  program. 

Because  that's  what  you're  really  judging  your 
success  fine. 

Judging  Success 

Program 

Success 

1 

I  think  that's  a  hard  question  to  answer  because 
there  are  so  many  unplanned  events  that 
happened.  An  example  would  be  this  current 
program.  There  is  a  we  have  in  place  a  plan  a 
detailed  plan  for  how  we  will  do  the  design 
work,  the  integration,  and  engineering  work, 
testing  ,  and  then  the  final  fielding  of  the  effort. 

Unplanned  events 

Challenges 

1 

When  something’s  outside  of  your  control  say 
Turkey  has  a  certain  home  indigenous  or  home 
developed  system  they  want  to  integrate  into 
the  aircraft  and  it's  key.  That's  its  own 
development  program,  separate  but  it  feeds  in 
and  can  impact  the  modification  program  if  it's 
behind  schedule.  So,  if  that  happens  your 
schedule  is  impacted.  Is  that  a  fault  to  the 

PBA?  No.  It's  really  an  outside  environment 
influence. 

Stable  user  needs 

Program 

Success 

1 

It  depends  on  your  contractor  base.  If  you  have 
a  very  experienced  contractor  and  a  large 
contractor  base  that  you  can  go  to  and  draw 
from  the  chance  of  PBA  working  is  much 
greater  than  if  you  have  a  very  fledgling,  early 
technological  base  or  a  base  that's  in  its  infancy 
or  its  new,  maybe  not  very  deep,  not  very 
experienced. 

Experienced 

contractor 

Program 

Success 

1 

I  think  the  environment  both  on  the 
government  side  and  on  the  industry  side  has  to 
be  ready  so  support  that.  And  I  think  if  early 
cooperation  between  the  government  and  the 
contractor  words,  a  joint  share  effort  and  that 
regard. 

Cooperation 

Program 

Success 
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Would  another  fonn  of  contracting  or  method 
of  contracting  work?  Yes  I  think  it  could.  It 
just  takes  a  lot  more  effort  and  I  think  the 
government  takes  risk  in  doing  that  as  long  as 
government  is  willing  to  do  that.  And  again  if 
they  don’t  judge  the  industrial  bases  ready  or 
capable  of  really  providing  positive  benefit 
back  to  doing  PBA  maybe  that's  not  the  right 
choice. 

Different 
procurement  type 

Environment  of 
the  Contract 

2 

No  not  really.  It's  all  been  US  contracts.  I  mean 
from  a  standpoint  of  doing  FMS  for  fourteen 
years  I  have  been  interfaced  with  foreign 
companies  and  we  are  doing  that  currently  in 
our  program.  But  it  is  more  from  a  technical 
standpoint  than  a  contracting  standpoint.  So,  I 
guess  I  have  to  say  no. 

No  difference 

Environment  of 
the  Contract 

3 

I  think  it  depends  on  the  environment.  If  I  were 
to  go  to  pick  a  country  northern  Africa 
somewhere  where  they  really  had  no 
technological  base. 

Industrial  base 

Environment  of 
the  Contract 

3 

And  depending  on  what  you  are  going  to  do.  In 
terms  of  let’s  go  build  a  fighter  aircraft.  You 
know  depending  on  where  you  are  at,  there  are 
only  some  countries  having  the  industrial  base 
to  go  do  that. 

Industrial  base 

Environment  of 
the  Contract 

3 

Or  some  other  highly  complex  effort.  If  you're 
just  going  to  contract  for  trash  pickup  garbage 
pickup  or  some  service  like  that  and  of  course 
you  could  do  that  I  think  anywhere.  It  depends 
on  what  the  task  is.  If  it  is  very  non-complex 
task  road  repair  some  service  like  that.  Those 
are  things  that  people  do  all  over  the  world  in 
anywhere  and  probably  there's  enough  industry 
no  matter  how  where  the  culture  is  what  the 
culture  is  whether  it's  a  third  world  country  or 
first  world  country  that  you  can  go  and  do 
performance  based  acquisition.  There  would  be 
no  challenge  to  do  that.  I  am  not  sure  how 
much  benefit  there  is.  For  example  road  repair. 

Type  of  job 

Type  of 
Contract 
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The  more  complex  the  task  that  you  want  to  do 
based  on  the  industry.  If  you  are  going  to  go 
international  if  you're  going  to  do  PBA  and 
you  bidding  environment  is  allowed  to  be 
international  then  obviously  you  can  do  that. 
But  if  you  want  to  stay  national  I  think  it  would 
depend  on  the  level  of  the  development  of  the 
country  you're  in. 

Industrial  base 

Type  of 
Contract 

4 

Really  it  is  reaching  agreement  on  a  good 
statement  of  work.  Once  you  do  your  SOW 
making  sure  that's  clear,  what's  in  statement  of 
work  and  whatever  technical  documents 
support  that.  So  that  you  have  a  good  base. 
Once  you've  done  that  you  have  something 
really  tells  you  how  you  want  to  measure  your 
success. 

Statement  of  Work 

Contract 

development 

4 

I  think  making  that  transition  from  getting  a 
good  statement  of  work  to  having  good 
measurements  at  the  end  of  the  program  what 
are  your  test  objectives  and  have  you  met  them 
so  that  you  know  you  really  done  what  you 
started  to  do.  I  think  that's  probably  the  most 
critical. 

Achieving  goal 

Program 

Success 

4 

If  you  started  from  the  beginning  and  you  don't 
correctly  state  what  it  is,  you  know  your 
objectives  are. 

Stating  goal 

Program 

Success 

4 

If  you  aren't  able  to  craft  a  good,  a  contractor 
doesn’t  present  a  good  statement  of  work  and 
you  don’t  reach  a  clear  understanding  of  what 
the  tasks  are  and  needed  to  be  done.  You  are 
go  opening  yourself  up  to  later  on,  you're  going 
to  have  to  go  back  in  contractually  and  correct 
something.  It's  going  to  cost  you  more  money 
and  you  are  going  to  find  you  didn’t  do 
something. 

Establishing  teams 

Program 

Success 

4 

And  also  matching  what's  the  correct  contract 
type  of  vehicle.  Should  it  be  a  firm-fixed  price 
contract,  fixed-price  incentive  fee,  time 
materiel  cost  plus. 

Procurement  type 

Program 

Success 

4 

Y ou  need  to  know  what  your  requirement  is  what 
you  can  scope  and  define  and  how  that,  what  best 
works  for  both  the  government  does  and  the 
contractor  so  there  is  a  win-win  situation. 

Cooperation 

Program 

Success 
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For  example  say  you  are  going  to  do  very 
intensive  development  program  and  a  lot  of 
testing.  It's  going  to  be  scope  what  that  testing 
is,  how  long  it  will  take,  how  much  we  have  to 
do,  how  much  retest  is  going  to  accomplish.  So 
you  can’t  scope  the  test  you  don’t  want  to  do 
firm-fix  price  contract.  Because  is  going  to  say 

I  have  no  idea  of  what  is  this  going  to  take  and 
here’s  my  risk  and  they're  going  to  bid  a  lot  of 
dollars  to  do  that.  So  in  that  case  you'd  be 
smart  to  go  cost-plus  or  time  materiel.  The  risk 
wasn’t  all  on  the  contractor  and  you'd  pay 
reasonable  price,  you  pay  for  the  work  done. 

Sharing  risk 

Program 

Success 

4 

It's  more  of  a  general  thing,  and  that's  your 
working  relationship  with  the  in  customer  and 
the  contractor.  I  think  that's  very  important. 

Cooperation 

Program 

Success 

4 

But  again,  being  able  to  know  what  is  trying  to 
achieve  and  how  good  define  objectives  to  me 
so  that  you  can  measure. 

Defining 

performance 

metrics 

Program 

Success 

5 

It  was  a  development  of  an  emulator;  it  was  a 
piece  of  test  equipment.  That  would  take  care 
of  some  older  technology  and  we  were  trying 
to  develop  this  emulator.  The  development  of 
emulator  was  in  progress  and  we  had 
requirements  unfortunately  one  of  the 
engineers  unknown  to  the  program 
management  was  working  with  the  contractor 
and  adjusting  the  requirements.  You  had  a 
changing  requirements  base  and  that  resulted  in 
contractor  having  to  do  additional  work. 

Unstable  user  needs 

Challenges 

5 

And  since  it  wasn't  done  through  the  contracts 
it  ended  up  being  where  they  fault  the  claim 
against  the  government  and  won.  It  had  to  be 
paid.  That’s  not  really  performance  based 
acquisition  issue. 

Unstable  user  needs 

Challenges 

6 

It's  really  a  team  effort. 

Establishing  teams 

Program 

Success 

6 

I  think  a  lot  of  it  has  to  do  with  your 
relationships  and  your  communication  and 
level  of  trust  which  is  not  really  a  performance 
based  acquisition  thing. 

Communication 

Program 

Success 
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6 

A  PBA  gives  you  a  structure  an  environment 
that  can  help  you  succeed  but  really  when  it 
comes  right  down  to  it,  its  people  that  makes  it 
work.  If  you  don’t  have  competent  people  and 
you  don’t  have  people  that  have  integrity  and 
trust  and  then  you  won’t  succeed. 

Quality  of  the 
Acquisition  team 

Program 

Success 

6 

I  think  PBA  is  a  good  tool  to  use,  it  brings  a 
lot,  it  allows  a  lot  of  ideas  to  come  out  and 
compete  against  one  another  but  ultimately 
even  if  you  are  depending  on  the  people  you 
have,  even  if  you  give  them  the  worst 
contracting  tool  or  acquisition  process  they  still 
achieve  success,  they  still  going  to  go  out  and 
make  do  their  best  to  achieve  success. 

Quality  of  the 
Acquisition  team 

Program 

Success 

6 

I  think  PBA  can  help  achieve  success  and  get 
you  the  most  for  your  money  which  I  think  is 
what  we  are  supposed  to  do,  try  to  do.  But 
again  it  really  comes  down  to  your  team,  the 
customer,  the  government,  and  the  contractor. 
As  long  as  everybody  works  together  and  has 
good  communication  and  trust  I  think  you  are 
going  to  have  success  whether  you  are  using 
PBA  or  not. 

Communication 

Program 

Success 

7 

It  is  the  customer  happy.  The  customer  feels 
like  he  got  value  for  and  did  it  meet  his 
objectives.  To  me  I  think  that's  the  most 
important  goal. 

Customer 

satisfaction 

Program 

Success 

7 

It  is  again  did  I  get  what  I  want  it  and  did  I  get 
value  was  it  a  good  value,  and  did  I  get  it  on 
time.  You  know  if  you  are  in  the  military  you 
have  operational  need  date.  Did  I  meet  my 
operational  need  date? 

Product  Quality 

Program 

Success 

7 

In  my  position  I  feel  sort  of  you  know,  you  got 
to  keep  it  healthy  industrial  base,  you  got  to 
keep  customer  happy. 

Keeping  healthy 
relationship 

Program 

Success 

1 

We  had  a  list  of  different  qualities  that  we 
wanted  that  the  contractor  can  perform  to  as  far 
as  the  quality  of  their  work. 

Product  Quality 

Program 

Success 
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It  ranged  from  things  like  how  well  doing  their 
system  engineering,  how  well  they  were 
defining  their  requirements  and  breaking  those 
down,  how  well  they  were  doing  their  testing. 

If  they  had  laid  out  all  their  test  objectives 
upfront  how  accurately  they  had  tested  those 
objectives,  how  accurately  they  had  got  back 
and  updated  what  they  needed  to  update,  how 
many  deficiencies  that  they've  found  in  their 
design. 

Defining 

performance 

metrics 

PBA  metrics 

1 

So,  if  they  did  a  regular  set  of  testing  and 
found  no  deficiencies  that  was  obviously  much 
better  than  if  they  did  a  poor  job  of  testing  and 
found  no  deficiencies  so  that  kind  of  played 
into  it. 

Testing 

PBA  metrics 

1 

We  had  a  several  different  administrative 
categories  as  far  as  them  submitting  their 
documentation  on  time,  them  billing  us  on 
time,  their  accuracy  of  their  billing.  Then  we 
went  into  things  as  far  as  retention  of  the 
employees  on  the  company. 

Managing 

Administrative 

process 

PBA  metrics 

1 

It  was  a  software  program  so  we  didn’t  want  to 
have  a  bunch  of  turn  over  with  the  software 
programmers  who  want  to  them  the  contractor 
maintain  the  core  set  of  people  who  are  going 
to  carry  through. 

Using  software 
program 

PBA  metrics 

1 

Timeliness,  I  think  that  pretty  much  covers  it 
as  far  as  how  we  incentivize  them.  We  had 
basically  across  the  board.  Everything  you'd 
want  to  good  contractor  to  do. 

Experienced 

contractor 

PBA  metrics 

1 

System  engineering,  testing,  reporting,  things 
on  time,  managing  their  people.  Those  were  all 
different  items  that  we  evaluated  every  six 
months.  And  based  on  the  award  fee  on  how 
we  evaluated  them. 

Award  Fee 

PBA  metrics 

1 

It  ranged  from  anywhere  80  percent  of  the  pool 
all  the  way  up  to  the  nineties  depending  on 
how  they  were  performed  in  that  time  period. 

Performance 

PBA  metrics 
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Well  just  the  fact  that  we  had  award  fee  made 
them  listen  to  our  comments.  So  that  was  much 
more  effective  than  not  having  an  award  fee. 

To  be  able  to  say  hey  we've  got  seven  percent 
of  profit  every  six  months  that  you  could  earn 
all  of  seven  percent,  we  are  going  to  deviate  a 
chunk  of  that  to  you  based  on  your 
performance.  That  helped  just  overall  as 
opposed  to  not  having  that  they're  being  a  fixed 
profit. 

Award  Fee 

PBA  metrics 

1 

What  worked  really  well  were  the  objective 
things  that  we  could  make  them  change.  If  we 
could  say  hey  your  staff  is  not  on  time  and  not 
accurate,  we  could  them  zero  for  the  specific 
area  that  fed  into  the  award  fee.  Those  areas 
had  a  lot  more  weight.  Because  they  knew 
when  they're  late,  they  knew  something  wasn’t 
right. 

Flexibility 

Program 

Success 

1 

The  things  didn't  work  so  well  were  the  really 
ambiguous  kind  of  fuzzy  measures.  You're 
doing  systems  engineering  ok  we’d  like  you  do 
it  better,  and  we  didn’t  have  a  concrete  reason 
of  what  was  better.  You  know  was  better  doing 
it  faster,  was  better  doing  accurately  we  just 
kind  of  said.  We  don’t  think  it  is  as  good  as 
we’d  like  it  to  be.  Those  kinds  of  things  we 
gave  them  feedback  they  said  we  don't  know 
what  to  do  with  that.  We  don’t  understand  how 
you  want  us  do  it  better. 

Fuzzy  measures 

Program 

Success 

1 

So,  the  more  objective  based  our  criteria  was 
the  easier  for  us  to  say  hey  you  are  not  getting 
the  stuff  on  time,  you  are  not  programming 
well  because  we  can't  see  the  amount  of 
deficiencies  that  you  have  or  you  have  lost  five 
programs  that's  not  what  we  want.  Those  were 
very  easy  to  score  them  on  and  communicate 
that  back  to  them  and  say  you  are  not  doing  on 
these  areas. 

Scoring  contractor 

PBA  metrics 

1 

The  areas  that  were  very  subjective  we’re  going 
to  just  had  to  make  call  if  they  were  doing  well 
or  not  well. 

Subjective 

Measures 

Challenges 

1 

I  would  say  slightly.  It  certainly  made  the  core 
group  of  leadership  at  the  contractor  performs 
better. 

Experienced 

contractor 

Program 

Success 
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I  think  it  performed  better  than  a  fixed  price 
contract  of  the  same  sort  of  magnitude.  Cost 
probably  not  because  in  the  end  we  actually 
paid  them  more  profit  probably  than  we  could 
squeeze  that  of  them  with  a  fixed  price 
contract. 

Cost 

Challenges 

1 

I  think  we  certainly  got  more  perfonnance  out 
of  them  first  of  all  we  are  cost  based  anyway 
we  are  paying  their  cost  additional  and  fee  at 
the  end.  So  perfonnance  wise  I  am  positive 
that  we  got  a  better  product  from  them  because 
we  did  perfonnance  based.  I  certainly  think 
that  the  performance  based  ad  it  better  quality 
product  in  the  end. 

Product  Quality 

Program 

Success 

2 

In  one  of  my  units  we  had  a  contract  called 
eagle  vision  and  I  believe  prime  contractor  was 
French  on  that  program.  The  issues  at  least  I 
heard  from  those  folks  with  that  contractor  was 
they  were  a  little  bit  laid  back  on  when  stuff 
was  going  to  get  done.  American  contractors 
tend  to  laid  out  schedule  and  then  we  the 
government  kind  of  beat  on  them  and  tell  they 
deliver  to  that  schedule  or  close  to  it.  The 
French  contractor  seemed  much  more  laid  back 
as  far  as  the  time  went. 

Work  ethics 

Culture 

2 

I  certainly  believe  that  having  performance 
elements  in  all  cases  incentivizes  the  contractor 
to  meet  the  requirements  based  on  they're 
going  to  make  more  money  if  that's  how 
they're  incentivized. 

Way  of  giving 
Incentive 

Program 

Success 

3 

Yeah,  if  I  have  the  choice  on  a  contract  I 
certainly  would.  If  I  had  to  do  it  differently  I 
would  tie  my  performance  based  criteria  to 
specific  milestones.  As  far  as  you  complete 
your  critical  design  review  then  we  evaluate 
your  perfonnance  and  give  you  sort  of  award. 

Defining 

performance 

metrics 

Type  of 
Contract 

3 

You've  completed  your  first  phase  of  testing, 
instead  of  just  doing  a  calendar  based  where  is 
every  six  months  we  just  hand  them  a  check 
based  on  how  we  did,  I  would  explicitly  tie  that 
to  the  performance  of  a  certain  event. 

Defining 

performance 

metrics 

Program 

Success 
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PBA  in  a  non-US  environment.  I  would  say 
yes.  If  it  was  up  to  me  I  think  that  there  is 
probably  one  of  the  better  methods  to 
incentivize  a  delivery  that  meets  your  cost 
schedule  and  perfonnance. 

Performance 

Environment  of 
the  Contract 

4 

The  hardest  part  was  the  subjective  measures 
of  whether  things  like:  program  management, 
system  engineering  was  good  or  bad,  or  how 
good  or  bad  was  it. 

Subjective 

Measures 

Challenges 

4 

We  had  laid  out  an  award  fee,  the  criteria  that 
makes  it  good  or  makes  it  bad.  And  it  is  very 
difficult  to  actually  pick.  If  say  there  is  five 
criteria  to  pick  like  your  perfonnance 
contractor  fits  in  number  four  bin  instead  of  the 
number  five.  What  ended  happening  for  us  we 
picked  certain  things  that  went  wrong  and 
certain  things  that  went  right  and  we  count 
those  weight  against  each  other  but  the  rest  of 
the  stuff  was  just  kind  of  was  ok. 

Defining 

performance 

metrics 

Program 

Success 

4 

So,  when  you  are  ready  to  do  your  award  fee 
and  make  your  evaluation  all  the  errors  that 
you  may  have  found  or  good  stuff  for  that 
matter  is  very  refreshing  your  mind  two 
months  ago,  but  something  happened  six 
months  ago  you  may  have  overlooked  it,  or 
you  may  forgot  about  it.  We  tried  to  do  was 
continuously  updated  the  database  with 
specific  comments  we  actually  created  an 
access  database  that  all  the  members  of  the 
program  office  if  they  noted  something  bad  or 
good  or  something  needed  changing, 
encourage  them  go  online  and  type  that  in  there 
so  that  we  had  a  record  of  whole  six  months. 

Scoring  contractor 
timely 

PBA  metrics 

4 

The  hardest  part  was  the  administration  of  the 
whole  thing  and  then  ensuring  the  consistency 
across  the  comments  and  then  being  fair  to  the 
contractor.  The  difficult  part  was  really  the 
subjective  nature  of  evaluating  good 
performance  versus  bad  performance.  If  I  had 
to  do  it  again  I’d  made  very  objective  criteria  of 
meeting  certain  design  review  successfully, 
completing  a  certain  task  successfully,  i  think 
that  would  eliminate  a  lot  of  the  other  issues 
with  subjective  evaluations. 

Ensuring 
Consistency  of 
comments 

Challenges 
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In  that  program  that  I  was  just  referring  to,  we 
did  fail  operational  testing.  But  it  really  wasn't 
contractor's  fault;  it  was  the  test  agency's  fault. 
They  didn’t  have  an  updated  software  on  their 
computers.  What  ended  up  happening  was  they 
basically  claimed  the  contractor  did  a  bad  job 
their  software  system  didn’t  work.  Turns  out 
the  test  agency  an  Air  Force  govermnent 
agency  had  old  software.  If  they  would  have 
the  most  recent  update  it  would  have  been  a 
little  different  result  on  the  back  side  of  the 
testing. 

Technical 

Difficulties 

Challenges 

5 

I  guess  the  other  horror  story  specifically 
related  to  the  performance  based  was  is;  we 
had  contracting  officer  who  was  very  black  and 
white  with  those  criteria.  So,  if  they  had  one 
issue  that  was  poor  or  they  were  late  on  one 
document  his  idea  was  we’ll  give  them  zero  for 
that  area.  Which  made  the  award  fee  go  from 
mid  nineties  down  to  eighty  one  which  put  the 
contractor  up  on  arms.  So  there  was  a  lot  of 
damage  control  that  we  had  to  do  with  that. 

Unfair  scoring 

Challenges 

6 

I  think  we  were  that  program  was  pretty  good 
in  the  fact  that  everybody  at  least  on  the 
government  side  was  a  pretty  good  team.  So, 
no  one  had  to  do  extraordinary  effort  to  get 
anything  done. 

Establishing  teams 

Program 

Success 

7 

I  think  the  primary  successful  criteria  are  that 
you  deliver  your  product;  it  performs  as 
intended  within  a  reasonable  amount  of  time. 

Product  Quality 

Program 

Success 

7 

The  hard  part  with  at  least  with  the  American 
acquisitions  is  that  in  my  opinion  we  have 
under  budgeted  every  program. 

Funding 

Challenges 

7 

I  think  you  got  to  deliver  something  that  works 
in  the  end. 

Product  Quality 

Program 

Success 

7 

And  if  you  do  that  with  reasonable  slips  in 
schedule  and  cost  I  think  that  is  a  successful 
acquisition. 

Schedule 

Program 

Success 
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Q# 

SENTENCE 

SUBCATEGORY 

CATEGORY 

1 

The  methods  we  used  from  the  program  office; 
we  worked  with  the  contractor,  we  give  them, 
they  had  the  authority  and  ability  to  choose  the 
best  methods  but  as  a  government  we  had  a 
better  idea  what  users  actually  want  it  so  we 
worked  with  them  to  help  find  the  best 
solution.  The  success  we  have  individual 
people  interactions  between  us  and  the 
contractor  as  opposed  to  any  contractual 
methods  that  we  used. 

Communication 

Contract 

development 

1 

The  biggest  thing  that  seemed  to  work  was  we 
on  the  government  side  needed  to  understand 
what  things  needed  to  be  done  and  we  let  the 
contractor  determine  how  they're  going  to 
them.  We  were  confident  in  their  methods  as 
well. 

Flexibility 

Contract 

development 

1 

The  program  was  a  very  long  one.  In  the 
beginning  phases  of  it  the  government  wasn’t 
as  involved  in  what  the  contractor  was  doing  to 
set  up  the  program.  So  we  had  a  problem  in  the 
development  of  the  software  and  that  they 
weren't  using  the  correct  metrics  to  detennine 
if  the  software  was  good  enough  to  use. 

PBA  metrics 

Program 

Success 

1 

The  government  needs  to  understand  what  the 
contractor  is  doing  so  they  can  recognize  if 
something  is  not  working  or  not  going  to  work. 
From  a  perfonnance  based  of  judging  or 
grading  if  the  contractor  on  how  well  they 
perform  you  definitely  need  to  have  some 
insight  along  the  way. 

Communication 

Program 

Success 

1 

I  have  seen  some  shorter  programs  where  that 
did.  On  the  larger  programs  there  are  so  many 
variables  that  cause  the  schedule  and  stuff  to 
slip  and  the  cost  go  up. 

Program  Length 

Type  of 
Contract 

2 

I  think  the  biggest  thing  with  non-US  contracts 
is  the  rules  and  regulations.  I  guess  there  would 
be  no  difference  non-US  contractor  and  US 
contractor.  There  may  be  legal  things  a  little  bit 
different  but  from  a  performance  based 
acquisition  I  don’t  think  there  would  be  any 
difference. 

Regulations 

Environment  of 
the  Contract 
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Q# 

SENTENCE 

SUBCATEGORY 

CATEGORY 

3 

If  I  have  the  opportunity,  yeah.  If  you  look  at 
non-government  contracts  just  in  a  free  market 
almost  is  everything  is  performance  based.  If 
you  perfonn  and  do  a  good  job  or  build  a  good 
product  you  are  rewarded  by  profit  or  future 
contract.  That  idea  should  work  very  well. 

Performance 

Environment  of 
the  Contract 

3 

Where  the  contractor  is  shouldn’t  make  a 
difference.  If  you  set  up  your  contract  of,  you 
know,  we  will  pay  you  for  the  work  you  do, 
based  on  your  performance  I  don't  see 
difference.  I  see  that  is  kind  of  international 
benefit  for  all  companies. 

Performance 

Environment  of 
the  Contract 

4 

I  don’t  think  the  government  understood  what 
exactly  the  contractor  was  doing.  We  didn’t 
keep  enough  communication  or  involvement 
with  them  along  the  process  to  ensure  them 
what  was  coming  out  of  their  perfonnance  was 
what  we’re  expecting. 

Communication 

Program 

Success 

5 

It  was  a  year  and  a  half  project  of  trying  to  get 
the  right  thing  for  the  people  who  were  going 
to  use  it  and  come  to  find  out  this  is  what  they 
want  it  at  all.  There  is  a  lot  of  good  work  that 
went  into  getting  them  but  it  just  wasn't  what 
they  wanted. 

Communication 

Program 

Success 

6 

When  we  finally  started  talking  to  the  users 
and  figuring  out  what  it  was  that  they  wanted 
we  went  to  go  talk  to  contractor  and  say:  look 
let  you  talk  to  the  users  so  they  can  tell  you 
exactly  what  they  want  it.  We  were  able  to 
change  some  of  the  stuff  so  the  users  did  get 
what  they  wanted. 

Defining  User 
needs 

Program 

Success 

6 

The  best  metric  would  be  a  combination  of 
getting  the  right  thing  that  you  need  in  a  short 
amount  of  time. 

Time 

Program 

Success 

7 

The  system  you  are  building  what  the  actual 
user  wants  or  whatever  it  is  you  are  getting  for 
the  users  what  they  want. 

Customer 

satisfaction 

Program 

Success 
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Q# 

SENTENCE 

SUBCATEGORY 

CATEGORY 

1 

Because  PBA  was  a  new  innovative  way  of 
buying  aircraft  for  the  Royal  Air  Force, 
traditionally  we  would  have  bought  a  number 
of  aircrafts.  And  manufacturer  of  the  aircraft 
tells  us  in  the  military,  you  have  bought  one 
hundred  aircrafts,  you  want  to  fly  those 
aircrafts  maybe  2000  hours  per  year,  and  this  is 
the  way  you  want  to  fly  them.  Based  on  our 
reliability  calculations  you'll  consume  this 
package  of  spare  parts.  That's  what  you  need  to 
buy. 

Dependent  to 
contractor 

Traditional 

Acquisitions 

1 

In  the  seventies  we  have  bought  an  aircraft 
called  Tornado  and  we  bought  a  very  large 
package  of  spares.  We  changed  the  way 
aircraft  flew  within  the  life  of  the  aircraft  so  we 
began  consuming  different  types  of  spares.  As 
a  consequence  we  ended  up  nearly  billion 
pounds  worth  of  spare  parts  in  the  warehouse 
not  adding  any  availability  to  the  aircraft's 
performance. 

Too  many  Spares 

Traditional 

Acquisitions 

1 

When  we  bought  Typhoon  we  decided  we 
weren't  going  to  buy  any  spares  at  all.  We 
would  just  buy  the  aircraft.  We  modeled  the 
consumption  of  the  aircraft  based  on  the 
reliability  figures  the  manufacturer  provided. 
Through  to  the  manufacturer  the  spares 
provisioning  was  a  lot  lower  than  we  had  ever 
seen  in  history. 

Collaboration 

PBA  metrics 

1 

We  integrated  the  contractor,  the  manufacturer 
into  the  supply  network  within  the  military 
which  is  the  first  time  that  was  ever  done. 

Cooperation 

PBA  metrics 

1 

So  we  derisked  the  military  responsibility  for 
providing  spares  support  to  the  aircraft 
initially. 

Sharing  risk 

PBA  metrics 

1 

So  the  manufacturer  not  only  was  he 
responsible  for  making  the  aircraft  and 
delivering  it  into  service  but  he  was 
responsible  supporting  it  and  justifying  the 
reliability  figures  that  he  had  predicted  would 
be  associated  with  the  aircraft. 

Cooperation 

PBA  metrics 
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SENTENCE 

SUBCATEGORY 

CATEGORY 

i 

We  decided  that  as  a  performance  based 
measure  we  could  not  afford  to  have 
everything  that  we  wanted  to  have.  Because  we 
had  not  got  the  funding.  If  we  could  we  would 
give  everything  to  the  contractor  and  they 
would  provide  one  hundred  percent 
functionality  and  availability  for  the  aircraft. 

Funding 

Challenges 

i 

We  had  performance  based  metrics  that  would 
pay  the  contractor  a  monthly  fee  to  provide  that 
level  of  availability  and  we  would  measure  the 
repair  cycle,  the  delivery  cycle,  downtime  in 
the  servicing  hangar,  we  would  measure  the 
robes  of  aircraft. 

Award  Fee 

PBA  metrics 

i 

That  acquisition  process  (Typhoon)  was  a 
multinational  acquisition  process.  We  had  very 
expensive  logistics  network  moving  parts 
around  all  of  Europe  (UK,  Spain,  Gennany)  to 
build  an  aircraft. 

Expensive  logistics 
network 

Traditional 

Acquisitions 

i 

And  limited  availability  when  it  came  to  spare 
parts.  Between  the  three  nations  we  decided  to 
put  spare  engines  in  different  centers  in  those 
different  countries.  That  was  too  expensive. 

Fimited  availability 

Traditional 

Acquisitions 

i 

Then  we  decided  to  put  spare  engines  in  only 
one  country,  Munich  in  Germany.  That  sounds 
good  in  theory.  They  just  couldn’t  afford  it. 

The  manufacturing  over  cost,  overrun.  The 
aircrafts  started  of  being  thirty  million  dollars 
each  and  with  overruns,  delays,  and 
modifications  each  aircraft  then  became  fifty 
million  dollars  each. 

Over  cost 

Traditional 

Acquisitions 

i 

And  really  everybody  was  watching  everybody 
else  to  make  sure  that  we  all  had  a  fair  share  of 
a  central  pool  of  spares.  The  thing  became  very 
complicated  and  to  measure  anything  within  in 
that  environment  performance  metrics  became 
almost  impossible.  It  was  too  complicated  to 
track  and  you  couldn’t  measure  anything. 
Because  nobody  could  decide  on  what  the 
actual  performance  metrics  would  be. 

Defining 

performance 

metrics 

Challenges 

i 

In  the  end  we  decided  that  the  best 
performance  metrics  we  could  have  would  be 
let  the  manufacturers  supply  the  engine  and  we 
would  pay  that  service. 

Perfonnance  Based 
Acquisition 

Type  of 
Contract 
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Q# 

SENTENCE 

SUBCATEGORY 

CATEGORY 

1 

If  all  of  those  criteria  are  fulfilled  and  you  get  a 
good  product  and  along  the  way  you  can 
measure,  you  can  gage,  you  can  watch  every 
stage  of  the  process,  high  the  performance  of 
the  project  actually  working  against  the 
baseline  then  that's  great.  That's  to  me 
performance  based  acquisition.  It's  identifiable, 
measurable,  and  successful. 

Measurable 

performance 

Program 

Success 

1 

There  can  be  increased  cost  with  performance 
based  logistics  with  regards  to  it  can  be  more 
manpower  intensive  to  monitor  and  check. 

More  Manpower 

Challenges 

1 

The  product  will  be  the  same  price  but  the 
management  of  the  process  and  monitoring  the 
process  and  watching  and  putting  a  team  to 
make  sure  you're  getting  what  you  want,  when 
you  wanted.  The  administration  of  the  project 
maybe  more  costly.  But  it  is  a  small  trade  off 
to  get  the  correct  product  within  the  correct 
time  at  the  correct  price.  That  is  the  way  we 
look  at  performance  based  metrics. 

Cost 

Challenges 

1 

It  is  hard  work  to  do  but  it  is  essential  to  do  it 
to  get  what  you  need  when  you  wanted. 

Schedule 

PBA  metrics 

2 

No.  The  reason  being  is  we  have  licensed 
contract  officers.  I  am  a  logistician.  No  I  don’t 
expect  any  differences  because  we  apply  the 
international  standards. 

International 

Standards 

Environment  of 
the  Contract 

3 

If  I  was  on  an  acquisition  team,  those 
performance  based  metrics  would  definitely  be 
the  part  of  the  contract. 

Perfonnance  Based 
Acquisition 

Type  of 
Contract 

3 

No.  The  only  criteria,  advantages  or 
disadvantages  when  we  look  at  any  major 
system  if  we  don’t  want  to  make  it  ourselves 
because  they  associated  with  cost  or  Turkey  or 
US  have  developed  the  system  that  is  so  good. 
We  decide  we  want  to  buy  it.  As  long  as  they 
meet  the  international  or  European  standards  of 
support  and  manufacturing  then  No. 

International 

Standards 

Environment  of 
the  Contract 

3 

I  mean  there  is  a  language  barrier  but  within 
the  contract  you'll  define  what  language  you're 
going  to  convey  them.  All  the  contracts  will  be 
in  English  not  Turkish  because  we  can't  read 
Turkish.  And  all  the  technical  publications  will 
be  English. 

Language 

Culture 
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SENTENCE 

SUBCATEGORY 

CATEGORY 

3 

So  there  are  no  advantages  or  disadvantage  as 
long  as  everybody  employs  the  same  standards 
that  we  work  to. 

No  difference 

Environment  of 
the  Contract 

4 

Money.  Overspent.  Always  a  problem.  We 
never  get  enough  money  from  the  government 
for  the  full  acquisition  process. 

Funding 

Challenges 

4 

As  a  military  we  are  not  skilled  enough  at 
identifying  all  the  associated  costs.  We  know 
how  much  the  aircraft  cost.  Aircrew,  training, 
synthetic  trainer,  the  simulator,  initial 
provisioning  for  the  spares,  we  can  map  as 
much  as  we  know  but  there  is  always  overruns, 
delays,  increased  costs  in  manufacturing. 

Establishing  teams 

Challenges 

4 

The  government  won’t  give  us  any  more 
money.  Because  they  want  to  use  it  on  health, 
education,  roads  which  are  what  the  public 
want.  The  public  don't  want  defense. 

Public  Desire 

Funding 

4 

Funding  is  always  a  major  issue. 

Funding 

Challenges 

4 

Quality  of  the  team  involved  in  that  acquisition 
is  always  a  major  success  factor. 

Establishing  teams 

Program 

Success 

4 

Plus  identifying  the  correct  performance 
metrics.  You  have  to  be  very  clever  in  getting 
the  right  metrics  in  that  project  and  having  a 
very  knowledgeable  skilled  team.  Those  are 
the  sort  of  critical  success  factors  that  you  have 
enough  leverage  within  your  perfonnance 
metrics  enough  leverage  over  the  manufacturer 
to  make  him  want  to  deliver  the  right  thing  at 
the  right  time. 

Defining 

performance 

metrics 

Program 

Success 

4 

If  there  is  no  cost  penalty,  no  time  penalty  to 
the  manufacturer  he  doesn't  care.  He's  still 
going  to  get  his  money.  But  if  there  is  a  clever 
performance  metric  that  the  manufacturer  will 
deliver  the  right  item  at  the  right  time. 

Penalty  as  Incentive 

Program 

Success 

5 

Only  that  one  I  explained  to  you  about  the 
engines.  We  spent  two  hundred  million  dollars 
on  one  hundred  engines  that  didn't  work  to 
begin  with,  they  were  eighteen  months  late  in 
delivery  to  the  air  force. 

Delivery  Time 

Program 

Success 
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5 

When  they  were  delivered  they  said  the  aircraft 
don’t  fire  because  they  were  so  hot,  because 
they  hadn’t  been  engineered  properly  and  we 
only  used  them  for  one  year  and  we  sold  them 
to  another  country. 

Product  Quality 

Program 

Success 

6 

To  be  honest  No.  I  don't.  Not  that  there  aren’t  it 
is  only  because  I  am  not  aware  of  they  are. 
Because  I  haven’t  been  involved  in  projects  for 
five  years. 

No 

Success  Story 

7 

I  think  for  the  really  public,  for  the  taxpayer  is 
that  you've  got  exactly  what  you  set  to  buy  in 
the  right  quantity,  to  the  right  standard,  it's  a 
quality  product,  and  it  was  within  the  time  that 
you  ask  for  it,  and  it  was  within  the  cost.  If  you 
take  those  factors  into  consideration  time,  cost, 
and  quality  as  long  as  they  are  met  and 
reliability  is  also  a  big  factor. 

Reliability 

Program 

Success 

7 

And  It  is  supportable  as  well  through  the  life  of 
the  equipment;  you  know  we  have  the 
manufacturing  base  in  industry.  The  company 
hasn't  gone  broke,  it  is  still  there.  So,  if  we 
want  to  buy  more,  or  modify  it,  or  develop  it, 
or  support  it  with  extra  spares  that  company  is 
strong  enough  to  be  there  for  thirty  years  of  life 
that  was  predicted  against  that  equipment. 

Supportability 

Program 

Success 
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Appendix  C.  Subcategory  Count  in  Each  Category 


SUBCATEGORIES 

CATEGORIES 

COUNT 

Communication 

Program  Success 

15 

Product  Quality 

Program  Success 

14 

Customer  satisfaction 

Program  Success 

13 

Cost,  schedule  and  performance 

Program  Success 

11 

Funding 

Challenges 

10 

Establishing  teams 

Program  Success 

8 

Schedule 

Program  Success 

8 

Defining  performance  metrics 

Program  Success 

7 

Award  Fee 

PBA  metrics 

6 

Collaboration 

Program  Success 

6 

Experienced  contractor 

Program  Success 

6 

Flexibility 

Program  Success 

6 

Schedule 

Challenges 

5 

Work  ethics 

Culture 

5 

Cost 

Challenges 

4 

Defining  performance  metrics 

Challenges 

4 

Security 

Challenges 

4 

Defining  Requirements 

Program  Success 

4 

Funding 

Program  Success 

4 

PBA  metrics 

Program  Success 

4 

Statement  of  Work 

Contract  development 

3 

Performance  Based  Acquisition 

Environment  of  the  Contract 

3 

Sub-contracting 

Environment  of  the  Contract 

3 

Cooperation 

Program  Success 

3 

Sharing  risk 

Program  Success 

3 

Sustainment 

Program  Success 

3 

Time 

Program  Success 

3 

Training 

Program  Success 

3 

Defining  Requirements 

Challenges 

2 

Enough  historical  data 

Challenges 

2 

Exchanging  Information 

Challenges 

2 

Subjective  Measures 

Challenges 

2 

Technical  Difficulties 

Challenges 

2 

Unstable  user  needs 

Challenges 

2 
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SUBCATEGORIES 

CATEGORIES 

COUNT 

Defining  performance  metrics 

Contract  development 

2 

Flexibility 

Contract  development 

2 

Monthly  reviews 

Contract  development 

2 

Planning  contract 

Contract  development 

2 

Systems  requirement  document 

Contract  development 

2 

Different  procurement  type 

Environment  of  the  Contract 

2 

Industrial  base 

Environment  of  the  Contract 

2 

International  Standards 

Environment  of  the  Contract 

2 

Regulations 

Environment  of  the  Contract 

2 

Cooperation 

PBA  metrics 

2 

Earned  value  management  system 

PBA  metrics 

2 

Establishing  teams 

PBA  metrics 

2 

Performance 

PBA  metrics 

2 

Periodical  reviews 

PBA  metrics 

2 

Scoring  contractor  timely 

PBA  metrics 

2 

Advocacy 

Program  Success 

2 

Periodical  reports 

Program  Success 

2 

Program  Length 

Program  Success 

2 

Quality  of  the  Acquisition  team 

Program  Success 

2 

Stable  user  needs 

Program  Success 

2 

Performance  Based  Acquisition 

Type  of  Contract 

2 

Authority 

Challenges 

1 

Complex  contract  environment 

Challenges 

1 

Ensuring  Consistency  of  comments 

Challenges 

1 

Establishing  teams 

Challenges 

1 

Experienced  contractor 

Challenges 

1 

Facilities 

Challenges 

1 

Finding  Contractor 

Challenges 

1 

Funding  Discontinuity 

Challenges 

1 

Fuzzy  measures 

Challenges 

1 

High  level  pressure 

Challenges 

1 

Interface  Problems 

Challenges 

1 

Interpretation  of  contract 

Challenges 

1 

Language 

Challenges 

1 

Manpower  Quantity 

Challenges 

1 

Non-US  Contractor 

Challenges 

1 

Performance  deficiency 

Challenges 

1 
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SUBCATEGORIES 

CATEGORIES 

COUNT 

Resistance  to  Change 

Challenges 

1 

Technological  systems 

Challenges 

1 

Type  of  job 

Challenges 

1 

Unexpected  circumstances 

Challenges 

1 

Unfair  scoring 

Challenges 

1 

Unplanned  events 

Challenges 

1 

Vague  awarding 

Challenges 

1 

Variety  of  Service 

Challenges 

1 

Communication 

Contract  development 

1 

Contract  success 

Contract  development 

1 

Defining  Requirements 

Contract  development 

1 

Design  reviews 

Contract  development 

1 

Interface  control  documents 

Contract  development 

1 

PBA  metrics 

Contract  development 

1 

Performance  measures 

Contract  development 

1 

Periodical  reports 

Contract  development 

1 

Sharing  risk 

Contract  development 

1 

Statement  of  Objectives 

Contract  development 

1 

Stating  goal 

Contract  development 

1 

Technical  Aspects 

Contract  development 

1 

Cultural  Similarity 

Culture 

1 

Language 

Culture 

1 

Performance  Oriented 

Culture 

1 

Training 

Culture 

1 

Contracting  guidelines 

Environment  of  the  Contract 

1 

Different  Company 

Environment  of  the  Contract 

1 

FMS  contract 

Environment  of  the  Contract 

1 

Future  contracting  method 

Environment  of  the  Contract 

1 

Non-US  Contractor 

Environment  of  the  Contract 

1 

Performance  Oriented 

Environment  of  the  Contract 

1 

Security 

Environment  of  the  Contract 

1 

Tangible  benefit 

Environment  of  the  Contract 

1 

Non-US  contractor 

Funding 

1 

Public  Desire 

Funding 

1 

Collaboration 

PBA  metrics 

1 

Communication 

PBA  metrics 

1 

Defining  performance  metrics 

PBA  metrics 

1 
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SUBCATEGORIES 

CATEGORIES 

COUNT 

Difficult  to  measure 

PBA  metrics 

1 

Experienced  contractor 

PBA  metrics 

1 

Managing  Administrative  process 

PBA  metrics 

1 

Schedule 

PBA  metrics 

1 

Sharing  risk 

PBA  metrics 

1 

Testing 

PBA  metrics 

1 

Using  software  program 

PBA  metrics 

1 

Various  comments 

PBA  metrics 

1 

Authority 

Program  Success 

1 

Award  Fee 

Program  Success 

1 

Different  procurement  type 

Program  Success 

1 

Documentation 

Program  Success 

1 

Good  perfonnance  history 

Program  Success 

1 

Integration 

Program  Success 

1 

Judging  Success 

Program  Success 

1 

Keeping  healthy  relationship 

Program  Success 

1 

Management  Reserve 

Program  Success 

1 

Measurable  perfonnance 

Program  Success 

1 

Number  of  delivery 

Program  Success 

1 

Penalty  as  Incentive 

Program  Success 

1 

Politics 

Program  Success 

1 

Proactivity 

Program  Success 

1 

Procurement  type 

Program  Success 

1 

Qualified  contractor 

Program  Success 

1 

Quantity 

Program  Success 

1 

Regulations 

Program  Success 

1 

Reliability 

Program  Success 

1 

Routine  contract 

Program  Success 

1 

Stating  goal 

Program  Success 

1 

Successful  program 

Program  Success 

1 

Supportability 

Program  Success 

1 

Testing 

Program  Success 

1 

Usability,  Cost,  Schedule 

Program  Success 

1 

Variety  of  Service 

Program  Success 

1 

Way  of  giving  Incentive 

Program  Success 

1 

Dependent  to  contractor 

Traditional  Acquisitions 

1 

Expensive  logistics  network 

Traditional  Acquisitions 

1 
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SUBCATEGORIES 

CATEGORIES 

COUNT 
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In  recent  years,  governments  and  military  organizations  have  been  facing 
shrinking  budgets,  aging  equipment,  declining  weapon  system  and  parts  availability, 
increasing  obsolescence,  vanishing  vendors,  increasing  sustainment  costs.  Most  military 
systems  have  been  used  for  decades,  so  future  planning  of  defense  procurement  is 
critically  important.  Performance-based  acquisition  is  a  strategic  method  to  manage 
business  by  promoting  flexibility  and  innovation  and  creating  win/win  solutions  through 
effective  communication,  organizational  goal  alignment,  and  clear  accountability  among 
the  buyer,  seller(s),  and  subcontractors.  Performance-based  acquisition  (PBA)  is  an 
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acquisition  method  structured  around  the  results  as  opposed  to  the  work  to  be  performed. 
Performance-based  acquisition  is  a  results-oriented  acquisition  strategy  used  to  achieve 
innovative  solutions  in  agency  programs. 

Any  major  system’s  life  cycle  can  be  separated  into  two  major  milestones.  These 
are  system  acquisition  and  system  sustainment.  It  has  been  estimated  that  about  30 
percent  of  all  dollars  spent  are  used  to  acquire  the  system,  while  the  remaining  70  percent 
of  all  dollars  are  used  for  support.  For  continuous  support  and  life  cycle  sustainment, 
implementing  performance-based  acquisition  can  help  organizations  to  seize  more 
benefits  out  of  a  system’s  acquisition. 

In  this  century,  many  governments  are  transitioning  their  acquisition  strategy 
from  traditional  methods  to  perfonnance-based  methods.  In  2000,  the  U.S.  Department 
of  Defense  set  a  goal  that  a  minimum  of  50%  of  the  service  acquisitions  would  use 
performance-based  acquisition  methods  by  2005.  F-35’s  life-cycle  cost  is  expected  to  be 
about  20%  lower  when  compared  to  that  of  legacy  systems  such  as  F/A-16  and  F/A-18. 
Although  PBA  is  a  new  method  in  the  acquisitions  area,  many  countries  and  private 
sectors  have  implemented  this  procedure  and  started  to  use  it  in  their  logistics  systems. 
The  United  Kingdom’s  (U.K.)  Defense  Logistics  Organization  is  transitioning  to 
performance-based  contracting  support  for  availability  and  capability  for  the  sake  of 
seeking  achievement  of  established  effectiveness  levels.  The  U.K.’s  Royal  Air  Force  is 
expecting  to  reduce  the  costs  by  12%  for  the  outsourced  maintenance  of  E-3D  Sentry 
AWACS  aircraft.  European  Aeronautic  Defense  and  Space  Company  EADS  CASA  has 
a  performance-based  acquisition  relationship  with  many  South  American  countries 
including  Chile,  Ecuador,  Paraguay,  and  Brazil. 
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Applying  PBA  methods  in  those  Western  countries  have  shown  successful  results. 
Implementing  performance-based  acquisition  methods  in  other  non- Western  countries 
might  produce  different  results.  The  factors  that  mitigate  the  risk  of  implementing  PBA 
in  different  countries  are  critical  for  acquisition  professionals.  In  recent  years  there  have 
been  many  studies  about  creating  high  performance  environment  in  business  practices. 
Since  performance-based  acquisition  is  relational  and  interactive  compared  to  traditional 
acquisition  methods,  understanding  cultural  differences  and  the  factors  that  affect  the 
implementation  of  PBA  may  give  acquisition  professionals  insight  for  developing  and 
managing  the  PBA  environment  in  different  countries. 

This  study  produced  three  critical  factors:  goal  definition,  challenges,  and  success 
boosters.  Both  government  and  industry  would  benefit  from  implementing  PBA  by 
defining  the  goal  clearly,  managing  challenges  effectively,  and  operating  the  success 
boosters  accurately.  Understanding  the  similarities  and  differences  between  countries 
would  help  acquisition  professionals  to  implement  PBA  in  non-western  countries.  This 
study  confirmed  that  cultural  factors  could  play  at  least  a  minor  role  in  the  success  of 
performance-based  acquisitions.  Communication,  cooperation,  and  more  integrated 
business  relationships  could  mitigate  the  risk  of  cultural  challenges  affecting  the  success 
of  PBA.  This  study  supports  the  view  that  the  performance-based  acquisition  approach 
can  be  implemented  in  government  and  industry  for  the  incoming  acquisition  programs 
regardless  of  the  cultural  differences.  The  views  expressed  in  this  article  are  those  of  the 
author  and  do  not  reflect  the  official  policy  or  position  of  the  United  States  Air  Force, 
Department  of  Defense,  or  the  US  Government. 
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